SYSTEM FOR VISUALIZING OCT SIGNALS WITH REFLECTION ON A VIRTUAL SURFACE

Abstract

The present invention relates to a system (100) for visualizing OCT signals, comprising a display means (30) configured for time-resolved display of image data and a control unit (40), wherein the control unit (40) is configured to receive a time-resolved OCT signal (19) of a selected field of view (66) of a sample (65) from an OCT system (10), to determine a time-resolved OCT image (31) with at least one object (70) and with a virtual surface (60) based on the OCT signal (19), wherein a reflection (70') of the at least one object (70) of the OCT image (31) is determined on the virtual surface (60), and to control the display means (30) to display the time-resolved OCT image (31) on the display means (30). The present invention further relates to a corresponding method for visualizing OCT signals.

Description

Subject of the invention

[0001] The present invention relates to a system for visualizing OCT signals, in particular for visualizing time-resolved OCT signals in the form of time-resolved OCT images, containing at least one object and a virtual surface. The present invention further relates to a method for correspondingly visualizing OCT signals. Technological background

[0002] The use of technological aids is an integral part of modern medicine. Both imaging techniques and robotic systems are now used as routinely in surgery as they are in diagnostics. The use of imaging techniques enables the visualization and discrimination of a wide variety of structures within the patient, and the image data obtained from the patient can be advantageously used in diagnostics, as well as in therapeutic and surgical procedures.

[0003] For example, a surgeon can use patient image data to better plan a surgical procedure and also receive support during the procedure. Robotic visualization systems are used to assist surgeons in performing surgical procedures. These systems typically have at least one camera for capturing images of the surgical area, supported by a tripod with an articulated structure. The tripod allows the camera to be positioned relative to the subject through translational and / or rotational movements in order to capture images of a desired field of view (FOV) of the surgical area. The use of optical stereo cameras enables the acquisition of 3D image data.

[0004] In addition to capturing surface information of a desired field of view, for example, using reflected or backscattered visible light, methods now also exist for capturing depth information of the field of view. These methods include optical coherence tomography (OCT), which allows the three-dimensional microscopic imaging of optically transparent and / or reflective objects and thus the acquisition of volume images of the biological tissue in the viewed field of view. Optical coherence tomography (OCT) is essentially an interferometric method using broadband light with a short coherence length. Systems for acquiring OCT data therefore typically feature an interferometer and a broadband light source with a spectral width of more than 1% of the central wavelength.

[0005] OCT data can be acquired sequentially or in parallel. Sequential OCT data acquisition is achieved, for example, by splitting a low-coherence source light beam into a sample beam and a reference beam at a beam splitter. These beams are then sent through two arms of an interferometer. A movable reference mirror is positioned in the reference beam path, and the object to be examined is positioned in the object beam path. By moving the reference mirror, a path difference between the object and reference beams, and thus the measured depth, can be adjusted. Using a mirror in the object beam path, the object beam is scanned two-dimensionally across the sample, resulting in a three-dimensional scan of the sample.

[0006] In such a time domain OCT (TD OCT) acquisition, the spectral width of the light source Δλ corresponds to a coherence length L C by LC =λ* / Δλ. The axial resolution of an OCT system corresponds to the coherence length L C of the light used and describes the resolving power of objects that are separated along the optical axis by at least the coherence length. For example, a light source in the near-infrared range with a central wavelength of 800 nm and a spectral width of 80 nm has a coherence length of 7 µm, and an OCT system with such a source thus has an axial resolution of approximately 1-10 µm. The transverse resolution of an OCT system is determined by the optics used in the object beam path, in particular by the object lens that focuses the light onto the object under examination.

[0007] Sequential acquisition of OCT data is also possible in the frequency domain (frequency domain OCT - FD OCT), whereby a distinction is generally made between the use of a tunable source (swept source OCT) and the use of a dispersive detector (spectral domain OCT - SD OCT). In swept source OCT, the frequency of the excitation light source, e.g. a laser, is tuned, whereby a path difference between the sample and reference beam and thus the scanned sample depth can be varied even without a movable reference mirror. In SD OCT, a broadband light source is also used, but the frequency components of the interference signal are separated before detection, e.g. by an optical grating.

[0008] OCT can capture cross-sectional and volumetric data of biological tissue, significantly increasing the information content for a surgeon. Therefore, the integration of OCT into surgical microscopes is desirable in order to be able to display both video data of the surface of a desired field of view and depth and / or cross-sectional images of the field of view, for example, simultaneously and / or superimposed. Since surgeons are primarily accustomed to working with (sometimes three-dimensional) video image data, the use of superimposed two-dimensional OCT data or OCT images represents a change for them. This could overwhelm the surgeons when interpreting the OCT images and may even lead to misinterpretations, for example, if surgeons interpret the OCT images in the same way as video image data.

[0009] Furthermore, OCT images are generated from OCT signals computationally, for example, using volume rendering, ray tracing, and / or ray marching. These methods inherently offer more diverse image generation options than the methods or algorithms typically used to generate (partially three-dimensional) video image data from the image signals acquired by a surgical microscope. Therefore, methods for generating OCT images (or for visualizing OCT signals) must be developed that select from the diverse possibilities those that best meet the requirements of medical practice.

[0010] In particular, intraoperative optical coherence tomography, preferably using swept-source OCT (ss-iOCT), i.e., real-time volumetric imaging, enables intraoperative 3D imaging at near-video rates during surgical procedures such as ophthalmology. The primary method for displaying volumetric OCT is direct volume rendering (DVR), as it enables the visualization of surgical procedures in four dimensions (4D-OCT) from any viewing angle. This allows an optimal viewing angle to be selected for each surgical task. Conventionally, the surgical field is viewed from a top-down perspective, which is limited by the viewing access of the surgical microscope. Side views in 4D-OCT, on the other hand, complicate hand-eye coordination and increase the cognitive load for a surgeon due to viewing the procedure from an unusual angle.

[0011] DE 10 2015 226 669 A1 describes a method and system for outputting augmented reality (AR) information applied to medical devices and / or patient examinations. The AR information is displayed to the patient in their field of vision to explain examination results.

[0012] DE 10 2022 120 203 A1 describes a system and method for acquiring and visualizing OCT signals in real time for medical applications. This system acquires OCT data and displays it as time-resolved images in various views, including three-dimensional volume images. The system can be integrated into surgical microscopes and enables surgeons to have improved depth perception during operations.

[0013] DE 20 2020 004 674 U1 describes a device for displaying real and virtual content in the form of generating virtual content that replicates the physical properties of mirrors in order to provide additional perspectives that are advantageous in medical scenarios.

[0014] The object of the present invention is therefore to provide an improved system and an improved method for visualizing OCT signals, which overcomes or at least reduces the disadvantages of the prior art and simplifies the correct interpretation of OCT images for a surgeon or user. Description of the invention

[0015] The object of the invention is achieved by the subject matter of the independent patent claims. Preferred developments are the subject matter of the dependent claims.

[0016] A first aspect of the present disclosure relates to a system for visualizing OCT signals, in particular for visualizing such signals acquired by means of a medical device, for example a surgical microscope. The system according to the invention comprises a display means configured for the time-resolved display of image data. The display means is preferably one or more screens, for example at least one screen of a surgical microscope, a screen permanently installed in an operating room, or a head-mounted display (HMD), for example video glasses. The screen is preferably a 4K and / or 8K-capable screen and / or a 3D screen configured for stereoscopic display.

[0017] The system according to the present disclosure further comprises a control unit connected to the display means, in particular for one-way or two-way data transmission. The control unit is configured to receive a time-resolved OCT signal of a selected field of view of a sample from an OCT system. The sample is, for example, a patient's surgical site, in particular an eye in ophthalmological surgery. However, the sample can also be any other surgical site, such as brain tissue in neurosurgery, tissue in the ENT area in ENT surgery, or gums, tartar, or dental nerves in dental surgery. Likewise, it can be any other tissue or preparation (in vivo, in vitro, or in situ).The time-resolved OCT signal is preferably acquired by generating a light signal and partially directing it as a sample beam onto the sample by means of the interferometer and then superimposing it in the interferometer with a reference beam also generated from the light signal in order to generate an interference pattern.

[0018] The control unit of the system according to the present disclosure is further configured to determine, based on the received time-resolved OCT signal, a time-resolved OCT image having at least one object and having a virtual surface, wherein a reflection of the at least one object of the OCT image is determined on the virtual surface. In other words, a reflective B-scan is generated as a volume cross-section, i.e., as a selected surface in the OCT volume. The virtual surface thus integrates reflections from the object in the volume. The virtual surface, which is preferably also referred to as a reflective B-scan, is visualized within the OCT volume so as not to distract the surgeon's attention from the focal area. The reflection is preferably calculated in real time during volume raymarching and therefore, when applied to 4D OCT, shows a reflected view of the object.

[0019] The control unit of the system according to the present disclosure is further configured to display the acquired time-resolved OCT image on the display means. In other words, the control unit is configured to control the display means to display the time-resolved OCT image with the at least one object and with the virtual surface, as well as the reflection of the at least one object on the virtual surface, on the display means. The display is advantageously performed with the at least one object and with the virtual surface, as well as the reflection of the at least one object on the virtual surface. The virtual surface facilitates the interpretation of the OCT image; in particular, it enables a better understanding of distances between objects and surfaces within the OCT image.Surgeons typically work with a stereoscopic microscope, which allows them to view a surgical field exclusively from an overhead perspective and manipulate anatomical structures with submillimeter precision. In an effort to improve surgical visualization, optical coherence tomography has been integrated into surgical microscopes, enabling high-resolution, depth-resolved imaging. Advances in spiral scanning and swept-source OCT have even paved the way for real-time volumetric imaging, enabling 4D visualizations of anatomical structures and surgical instruments. In such systems, GPU-accelerated direct volume rendering (DVR) has proven to be an effective method for visualizing surgical maneuvers, enabling real-time presentation of 4D OCT data on stereo displays.Instead of viewing the surgical field from above, as with stereoscopic microscopes, the depth-resolved properties of 4D-OCT can be directly and fully utilized by providing oblique or even more extreme lateral views. This could even lead to more precise tool-tissue interactions, as visualizing the surgical field from alternative angles allows for improved distance perception. On the other hand, such arrangements also introduce new complexities, as surgical interaction from a lateral perspective is not common in current ophthalmic procedures. In particular, hand-eye coordination is challenged when navigating instruments to a target from an unusual perspective, placing greater mental demands on the surgeon.One example is the lateral perspective, which can be advantageous for grasping a membrane during retinal peeling, as it naturally reveals the distance between the surgical instrument and the retina or a retinal membrane. Although this view allows for more precise instrument maneuvering, navigating the instrument to a specific B-scan cross-section is more mentally demanding than with the traditional top-down view. This invention provides a form of virtual augmentation integrated with 4D OCT visualizations to reduce the surgeon's cognitive load and improve spatial understanding from arbitrary perspectives, allowing the potential of 4D OCT to be better utilized.In other words, displaying the time-resolved OCT image with the at least one object and the virtual surface, as well as the reflection of the at least one object on the virtual surface, on the display means represents an intelligent virtual B-scan mirror (IVBM) by integrating a virtual mirror into a selected volume cross-section in 4D OCT, representing a virtual B-scan. This provides a novel visualization concept for improving targeted instrument interaction in 4D OCT-guided surgery. Capturing an object arbitrarily positioned in space is challenging for the surgeon without any perceptual cues. However, the reflection provides an additional view that significantly facilitates navigation to the target.The system according to the present disclosure thus advantageously enables optimal visualization of the OCT signal according to the needs of the user, thereby optimally exploiting the potential of the OCT signal, providing the user with maximum information content, and avoiding misinterpretations.

[0020] The control unit is preferably designed to control an OCT system for acquiring a time-resolved OCT signal of a selected field of view (region of interest - ROI) of a sample, wherein the field of view is preferably selected by a user. Furthermore, the system according to the present disclosure preferably comprises an OCT system. The OCT system preferably comprises a light source, for example a broadband light source, designed to illuminate a sample. This light source is preferably a tunable laser (swept source), for example a broadband laser, a supercontinuum laser, and / or an ultrashort pulse laser. A tunable laser can be a narrowband light source at a given time, the center frequency of which, however, can be temporally variable in a targeted manner, or it can be formed from a plurality of narrowband light sources.However, other broadband sources can also be used, such as a superluminescent diode, for example in an FD-OCT.

[0021] The OCT system preferably further comprises an interferometer designed to generate and superimpose a sample beam and a reference beam, for example, a Michelson, Mach-Zehner, or Koster interferometer. The interferometer preferably comprises a beam splitter for generating and superimposing the sample and reference beams from the light of the broadband source, a reference beam path, and a sample beam path. Furthermore, the interferometer preferably comprises means for adjusting the sample depth under investigation. Depending on the measurement method, this may be a means for generating a path difference (such as a mirror movable within the reference beam in SD-OCT), a means for separating light with a specific path difference (such as an optical grating in FD-OCT), or a means for generating light with a specific path difference (such as a tunable source in swept-source OCT).

[0022] The OCT system further comprises a scanning mechanism configured to scan the sample with the sample beam. The scanning mechanism is particularly configured to scan the sample beam in two dimensions across the sample. The scanning mechanism is preferably a scanning mirror, but other scanning mechanisms, such as a fiber optic scanner, a prism scanner, a Palmer scanner, or the like, can also be used. In an OCT system configured for full-field OCT, a scanning mechanism is unnecessary.

[0023] The OCT system further includes a detector configured to detect an interference pattern generated by the superposition of the sample beam and the reference beam. The detector may be, for example, a line detector, a two-dimensional detector array, a photodetector, or a dispersive detector. The detector may be configured, for example, as a CCD or CMOS detector.

[0024] The OCT signal is preferably an interference signal, with the modulation of the interference signal's envelope encoding reflection properties of the sample. Using the scanning mechanism, the sample can be scanned two-dimensionally at a depth set by the path difference. A clock frequency (refresh rate) for the time-resolved OCT signal is determined based on the scanning mechanism used, the means used to select or generate the path difference—for example, an adjustable mirror in the reference beam, an optical grating in front of the detector, or a tunable broadband light source—and the detector's repetition rate.

[0025] The control unit computationally determines a time-resolved OCT image based on the OCT signal, preferably by means of volume raycasting, volume rendering, ray tracing and / or ray marching, wherein according to the invention a time-resolved OCT image is determined with the at least one object and with the virtual surface, wherein a reflection of the at least one object of the OCT image is determined on the virtual surface.

[0026] In a preferred embodiment of the system according to the present disclosure, the reflection of the at least one object on the virtual surface is determined when a ray emanating from an observation point and reflected by the virtual surface intersects the at least one object. For example, in ray casting in direct volume rendering, the reflection of the at least one object on the virtual surface is determined by emitting a ray reflected from the virtual surface in the direction of the at least one object, and when the ray emanating from the observation point and reflected from the virtual surface (i.e., a ray from an observation or viewing direction of the volume rendering) intersects the at least one object. The reflection is then generated when an intersection point exists between the ray reflected from the virtual surface and the at least one object on the virtual surface.

[0027] Preferably, the reflection of the at least one object on the virtual surface is only determined if the beam emanating from an observation point and reflected by the virtual surface intersects the at least one object within a predefined distance. In other words, the virtual surface is then only sensitive to an object or voxels associated with it that are located within a predefined proximity, for example, within a distance d, of the virtual surface and are connected to the object.

[0028] In a likewise preferred embodiment of the system according to the present disclosure, the virtual surface is formed by a plurality of voxels, and the time-resolved OCT image in the virtual surface is determined based on an OCT signal associated with voxels forming the virtual surface. For example, during volume rendering such as volume ray marching, voxels with a position described by three coordinate axes are integrated into the virtual surface if these voxels form the virtual surface. For this purpose, only components of the OCT signal associated with these voxels are used. Thus, it is precisely possible to form the virtual surface based on selected signals and to generate the time-resolved OCT image according to specific criteria. Furthermore, it is particularly easy to check whether and which voxels are integrated into the virtual surface and contribute to determining the time-resolved OCT image.

[0029] In a particularly preferred embodiment of the system according to the present disclosure, an OCT signal associated with voxels forming the virtual surface is preferably displayed in an enhanced manner in the OCT image. In other words, volume intensities integrated into the virtual surface are visually highlighted on the virtual surface of the OCT image. In other words, this is achieved by highlighting voxels integrated into the virtual surface, for example, in volume rendering. Preferably, target structures aligned with the virtual surface are also enhanced. These target structures preferably include, among others, anatomical target structures such as a macular hole, a location on the retina where peeling should be initiated, or a retinal membrane after peeling has been initiated.These are then particularly preferably displayed with even greater emphasis than the other voxels forming the virtual surface. The virtual surface can be described mathematically, so that it is particularly easy to check whether voxels are assigned to the virtual surface and additional target structures, and based on the result, the corresponding voxels are amplified in the OCT image. Preferably, an amplified display in the OCT image occurs by using a dynamically modifiable parameter to amplify the corresponding voxels and taking this into account in an overall appearance of the voxels forming the virtual surface. Furthermore, perceptual cues are enabled by visualizing and amplified an intersection point of the at least one object with the cross-section if the object or voxels assigned to the object are also assigned to the voxels forming the virtual surface.

[0030] Furthermore, the time-resolved OCT image on or in the virtual surface is preferably determined based on an OCT signal associated with voxels arranged behind the virtual surface in an observation direction. This preserves volume structures behind the plane in the OCT image. Voxels that are not contained in the virtual surface but are located behind it are rendered using direct volume rendering conventions known to those skilled in the art. This is preferably done using an adaptive opacity transfer function that is applied to voxels arranged behind the virtual surface in the observation direction. These corresponding voxels are preferably determined by checking whether a voxel is located behind the virtual surface along an observation direction. This preserves a three-dimensional view and does not impair the user's vision.Essential components that would otherwise be hidden by the virtual surface are still displayed, so that all important information is retained.

[0031] In a likewise preferred embodiment of the system according to the present disclosure, an OCT signal that is assigned to voxels arranged behind and / or in front of the virtual surface in a viewing direction is preferably displayed in an attenuated manner in the OCT image. In other words, the virtual surface is extended by a type of semi-transparency, whereby the background is preserved and the virtual surface does not cover it. The transparency value can assume values ​​between 0 and 100%, whereby a transparency is generated according to the specific selection for the OCT signal that is assigned to voxels arranged behind and / or in front of the virtual surface in the viewing direction. The degree of transparency is selected accordingly in order to create a desired visual appearance. In particular for surgical interventions below the retinal surface, such as, for example,For subretinal and intraretinal injections, the volume associated with voxels arranged in front of the virtual surface in the viewing direction is rendered semi-transparent by attenuation, so that the reflection of at least one object on the virtual surface is visible through the retina. Thus, the possibility of displaying additional useful information is expanded by the transparency of the virtual surface created by the attenuation, since, on the one hand, structures behind the surface remain visible, and, on the other hand, structures in front of the virtual surface are rendered transparent.

[0032] Preferably, the at least one object comprises a medical instrument and / or an anatomical structure. The virtual surface therefore functions as an intelligent mirror in that it is only sensitive to voxels associated with a medical instrument and / or an anatomical structure. By mirroring these objects, it is made considerably easier for the surgeon, for example, to detect and estimate distances and directions of movement. The mirroring of a medical instrument provides the surgeon with additional orientation aid without having to give up the top view of the field of view. This maintains the advantageous view for effective hand-eye coordination and, at the same time, enables, for example, improved perception of distances.The distance to a target, such as when approaching small structures at or above the retina during epiretinal membrane (ERM) or internal limiting membrane (ILM) peeling, can be visualized with minimal magnifications and without additional images or angles of view that would distract the surgeon's visual focus from the target area.

[0033] In a preferred embodiment of the system according to the present disclosure, the control unit is further configured to determine the at least one object by segmenting the OCT signal or the OCT image. The segmentation is carried out in particular before the generation of the at least one object, based on the OCT raw data and / or based on an OCT image determined therefrom. By segmenting the OCT signal, the control unit is advantageously able to identify the position of the at least one object in the OCT volume. Various algorithms for automatically segmenting objects in image data are known to those skilled in the art, and these algorithms are used in the present case by the control unit to determine the reflection of the at least one object.These algorithms include traditional threshold-based image processing algorithms as well as newer, machine learning-based 2D and 3D segmentation networks (e.g., a U-Net segmentation network). The control unit is preferably configured to obtain the segmentations from individual 2D OCT B-scans or directly from the entire 3D OCT volume. The positions obtained in this way can be advantageously used in volume rendering, ray tracing, and / or ray marching, as described above.

[0034] According to this preferred embodiment, one or more objects of interest are first partially or completely identified using a segmentation method. Mirroring is then performed only for the objects of interest.

[0035] Preferably, the segmentation involves a combination of a convolutional neural network and traditional image processing. This makes it possible to achieve real-time performance.

[0036] Particularly preferably, the OCT signal comprises a plurality of A-scans, and the control unit is configured to determine an en face view (top view) based on the plurality of A-scans, to determine the at least one object by segmentation in the en face view, to determine a plurality of A-scans of the plurality of A-scans containing the at least one object, and to determine a position of the at least one object in the depth direction in the plurality of A-scans. In other words, when determining the en face view, an integration of intensities along the A-scans is performed. Subsequently, the at least one object is then determined by segmentation in the en face view, preferably by comparing the integrated intensities with a threshold value, wherein the object is determined if the predefined threshold value is exceeded.This is preferably done during raymarching along a ray emanating from an observation point and reflected by the virtual surface, and checking for an intersection point. The threshold is preferably determined empirically to reject OCT speckle noise and then to determine whether the sampled point can be assigned to the object. In order to preferably only mirror the object when it is located near the virtual surface, the number of sampled steps along the reflected ray is limited. Thereafter, a plurality of A-scans of the plurality of A-scans are determined if they contain the at least one object. Furthermore, a position of the at least one object in the depth direction is determined in each A-scan of the plurality of A-scans. In other words, since the virtual surface acts as a mirror for the at least one object, mirror candidate voxels must be identified in the volume.To achieve the real-time processing rates required for 4D OCT visualizations, the instrument is preferably first identified in a 2D projection image of the volume. This creates a 2D projection image encoding the average intensity along each OCT A-scan. This image is then passed to a convolutional neural network, preferably a U-NET or a U-NET-like convolutional neural network, to generate a binary object map. Since the OCT signal is completely blocked at the object surface, for example, in the case of a medical instrument, anatomical structures below the instrument are obscured, and the corresponding A-scans contain only object-related voxels. The 3D position of the voxels associated with the object can therefore be determined from the detected A-scans in the binary object map.Before the OCT volume and the object map are forwarded to a volume rendering algorithm, the volume is preferably additionally processed by applying a 3D median filter to reduce speckle noise in the OCT.

[0037] Preferably, segmentation involves assigning voxels to the object. In addition to information about the position or location, as well as an intensity or a scatter value, voxels therefore contain information about an assignment to the object. This is preferably done in binary form, with a binary value confirming or denying an assignment. Preferably, the assignment is determined by comparing the intensity of a voxel with a predefined threshold. If this threshold is exceeded, for example, an assignment to the object is assumed.

[0038] The control unit according to the present disclosure is further preferably designed and configured to identify or segment the at least one object based on object-specific information. In other words, additional information characterizing the object can be used to identify and segment this object. This information includes, for example, information on the size, geometry, estimated relative position, and / or material of the object. The object-specific information further preferably also includes information on a surgical phase (a phase of a specific surgical procedure) in which the segmentation of the respective object is desired. In combination with a phase detection algorithm, variable segmentation of objects can thus be carried out fully automatically.

[0039] The system according to the present disclosure preferably further comprises an interface configured to detect user input. The interface is preferably a handset, a footswitch, and / or means for detecting head and / or eye movement or an actual viewing direction, for example, integrated into video glasses or a head-mounted display (HMD). The interface can further be configured to detect voice commands and may comprise at least one microphone for this purpose. Likewise preferably, the interface is a keyboard, a joystick, a mouse, a touchscreen, or a combination thereof.

[0040] According to this preferred embodiment, the control unit is further configured to control the interface for detecting a user input. Furthermore, the user input is preferably preceded by an input request, which is output to the user, for example, by the output means. Likewise, the user input is preferably a selection from a plurality of predefined input options, which, for example, specify a virtual illumination direction and / or an intervention to be performed. Likewise, the user input is preferably a user input that can be freely defined by the user within a predefined framework. Likewise, the control unit is preferably configured to assist the user in the user input, for example, by displaying a plurality of dialogs for the targeted query of specific user inputs.

[0041] According to this preferred embodiment, the control unit is further configured and designed to segment a plurality of objects in the OCT signal or the OCT image. In other words, no targeted segmentation takes place (for example, using object-specific information); instead, all objects in the OCT signal or OCT image are first segmented. The control unit is further configured to determine or select at least one object from the plurality of segmented objects based on a user input received via the interface. The mirroring can thus be created for only a single object of interest, for example, a surgical tool, but also for multiple objects of interest, for example, multiple surgical tools or anatomical structures.

[0042] In a likewise preferred embodiment of the system according to the present disclosure, the control unit is configured to determine a color coding of the reflection of the at least one object based on a distance between the at least one object and the virtual surface in the time-resolved OCT image. When integrating the reflection of the at least one object, a color space is therefore used to encode the distance between the reflected object and the virtual surface. Preferably, the distance is determined based on a ray emanating from an observation point and reflected by the virtual surface, which ray intersects the at least one object, and is encoded thereon using a color coding. Preferably, the color coding comprises the L*a*b* color space, preferably a perceptually linear color space.A color space is perceptually linear if a change in length in any direction X of the color space is perceived by a human as the same change. A non-perceptually linear color space can exhibit strong contrasts when transitioning from one hue to another. In data visualization, these contrasts can be mistakenly viewed as changes in the data rather than transitions in the color palette. In other words, a perceptually linear color space means that the same geometric distance at any point in the color space reflects the same amount of perceived color differences. This enables an improved spatial representation of at least one object. For example, the object is a medical tool, the spatial position of which can be classified and estimated much better using the color coding.

[0043] Furthermore, the control unit is preferably configured to vary a position and / or orientation of the observation point, the observation direction, and / or the virtual surface. As already explained, the rendering of 3D OCT volume images can lead to problems in the correct interpretation of spatial relationships of objects / surfaces in the OCT image. For example, the relative position of a surgical tool tip and an anatomical structure can be incorrectly interpreted depending on the viewing angle. This problem is solved by positioning a virtual surface. The virtual surface comprises the reflection of the at least one object, wherein both the virtual surface and the reflection thereon are dependent on the position and / or orientation of the observation point, the observation direction, and / or the virtual surface itself.By varying the position and / or orientation of the observation point, the observation direction, and / or the virtual surface, the position perception of objects of interest in the OCT image and the estimation of their distance from anatomical structures can be supported. The control unit is also preferably configured to display the position and / or orientation of the observation point, the observation direction, and / or the virtual surface in the time-resolved OCT image. This advantageously further supports the user's understanding of the reflection generated on the virtual surface.

[0044] According to this embodiment, the system according to the present disclosure preferably has an interface designed to detect user input. The interface is preferably the interface already described above. The control unit is then preferably configured to adjust the position and / or orientation of the observation point, the observation direction, and / or the virtual surface based on the user input. The position and / or orientation of the observation point, the observation direction, and / or the virtual surface can thus advantageously be adjusted manually in order to adapt them to the surgeon's field of view or focus area and to display a perspective particularly suitable for them. This allows the user to perform precisely those manipulations that they consider helpful for their personal understanding.

[0045] Alternatively or additionally, the system according to the present disclosure further comprises a device interface designed to detect a device parameter. The device interface is preferably an interface for connecting a surgical microscopic system, as described in detail below. Likewise preferably, the device interface is also an interface for connecting any other imaging system. The device interface can also be an interface for connecting a medical instrument. Likewise preferably, a system for tracking a medical or medical instrument can be connected via the device interface, wherein the tracking system can also be the surgical microscopic system or another imaging system.The medical instrument is, for example, a pointer, a probe, a pair of tweezers, an awl, a phaco tip, an endoscope, an endo-LED of a gripper, or the like. According to this embodiment, the control device is preferably configured to communicate with the device interface, in particular to communicate bidirectionally, and further to control the device interface to acquire a device parameter of a device connected via the interface.

[0046] According to this embodiment, the control unit is further configured to determine the position and / or orientation of the observation point, the observation direction and / or the virtual surface based on a device parameter detected by means of the device interface.

[0047] In a likewise preferred embodiment of the system according to the present disclosure, the at least one object further comprises a medical instrument. The device parameter preferably describes a location, a position, and / or a state of a connected medical instrument (or a tracked medical instrument) tracked by the tracking system. The medical instrument is, for example, a probe, a pointer, a pair of tweezers, an awl, a phaco tip, an endoscope, an endo-LED, or the like.

[0048] A type of medical instrument can preferably be identified using a device interface for connecting the medical instrument, by means of a tracking system (for example, by recognizing a target ID), and / or by input via a user interface. Likewise, the type of medical instrument introduced into the field of view of the surgical microscopic system is preferably identified by image analysis of the video image data, for example, by means of segmentation and object recognition. A position of the medical instrument is preferably determined based on the detection of a marker and / or a target comprising several markers, wherein the marker can be a marking on or a structure of the medical instrument.The detection of the marker and / or target is preferably performed using the surgical microscopic system and, if necessary, using additional light sources (e.g., infrared LEDs) and / or after registration / calibration of the medical instrument (e.g., by positioning a tip of the medical instrument at a defined location). The spatial position of the medical instrument can also be determined by the instrument itself. Advantageously, the position and / or orientation of the observation point, the observation direction, and / or the virtual surface can be automatically adjusted in this embodiment depending on the distance of the instrument from the ocular anatomy or other regions of interest in the OCT image.For example, the position and / or orientation of the observation point, the observation direction and / or the virtual surface can be arranged with a decreasing distance of the medical instrument to the virtual surface and / or, for example, an anatomical tissue, with a decreasing distance to these areas, so that an adaptation takes place according to the current situation or the current process and a suitable highlighting, transparent representation and advantageous mirroring for better spatial perception takes place for this situation.

[0049] The state of the medical instrument placed in the field of view of the surgical microscope system is preferably also determined based on image analysis of the video data. For example, the image data can be used to determine whether a pair of forceps is open or closed. Furthermore, a user input to change a state can be read by the control unit; for example, a user input to activate a phaco tip signals a change in its state. Furthermore, a sensor attached to the medical instrument can detect the change in its state, for example, the closing of a pair of forceps, and transmit a corresponding sensor signal to the control unit.A device parameter received via the device interface regarding the status of a connected medical device preferably also includes the actuation of an input device of the medical device, so that in this case the device parameter simultaneously represents a user input. Likewise, various device parameters are preferably considered together, such as the position of the medical instrument (for example, determined by the instrument itself and / or by a tracking system) and the actuation of an input device of the medical instrument. Thus, for example, a sequence of position and / or orientation of the observation point, the observation direction, and / or the virtual surface can advantageously be adjusted according to a sequence of positions of the instrument.A device parameter describing the medical or medical-technical instrument is received via the device interface from the instrument itself or from a tracking system.

[0050] According to this preferred embodiment, the control unit is further configured to determine the position and / or orientation of the observation point, the observation direction, and / or the virtual surface based on the position, type, and / or state of the medical instrument. Thus, the position and / or orientation of the observation point, the observation direction, and / or the virtual surface can be determined, for example, based on the position of a pointer or the state of a gripper. A change in the state of a medical instrument of a specific type and / or at a specific location is preferably indicative of a specific phase of an operation. Thus, based on the detection of the position, type, and / or state of the medical instrument, the optimal position and / or orientation of the observation point, the observation direction, and / or the virtual surface for this phase can be selected.This may also include adjusting other display parameters of the OCT images, such as a zoom level, a virtual viewing direction, a displayed depth, a cutting direction, etc.

[0051] Based on the aforementioned information, the control unit is, for example, able to determine automatically or based on a user input that a performed procedure is a membrane peeling using tweezers and, in this case, to further determine a distance or a position of the tweezers relative to the virtual surface and / or the eye and, depending on the aforementioned determinations, to adapt or set the position and / or orientation of the observation point, the observation direction and / or the virtual surface.In a further example, the control unit is preferably able to determine automatically or based on a user input that a performed procedure is a subretinal injection of a "bleb" or a placement of a retinal stent and, based thereon, to adapt the position and / or orientation of the observation point, the observation direction and / or the virtual surface in such a way that a user is supported in positioning the needle or the stent, for example by positioning the virtual surface with the shortest possible distance to the retina or a point of intervention and thus placing the reflection flush with the real intervention site so that a user can optimally assess when the needle is correctly placed.

[0052] Also preferably, according to this preferred embodiment, the control unit is configured to determine a target structure based on anatomical information and to adjust the position and / or orientation of the observation point, the observation direction, and / or the virtual surface based on the determined target structure. Known algorithms can be used to detect target structures based on anatomical information. These target structures include, among others, anatomical target structures such as a macular hole, a location on the retina at which peeling should be initiated, or a retinal membrane after peeling has been initiated. These are accordingly determined in order to then, for example, position and align the virtual surface in such a way that the time-resolved OCT image in the virtual surface is determined based on an OCT signal that is assigned to voxels forming the virtual surface and includes these target structures.In this case, the voxels forming the virtual surface also encompass the target structure and are preferably displayed in greater detail, so that this positioning of the virtual surface provides a particularly informative and helpful view.

[0053] In a preferred embodiment of the system according to the present disclosure, the virtual surface comprises a concave or convex surface. In the case of a concave virtual surface, this functions as a virtual concave mirror, also called a concave mirror. The resulting image depends significantly on the distance of the object from the concave mirror, which must be taken into account accordingly. If the object is located between the mirror and the focal point, a virtual, upright, and magnified image is created. Accordingly, the virtual surface on which the reflection of the at least one object is determined is preferably arranged such that the object is located between the virtual surface and the focal point. In the case of a convex virtual surface, this functions as a virtual convex mirror. The focal point is located behind the virtual surface. A virtual (apparent), upright, reduced image is created.Accordingly, a reduced image is created, with a larger viewing angle. This is preferable if you want to see a large part of the surrounding field of view.

[0054] Furthermore, the at least one object and / or the reflection of the at least one object is preferably detected pointwise and / or schematically in the time-resolved OCT image. Accordingly, key points or areas and / or an outline of the object are detected and displayed. This makes it possible to display only relevant areas of the object and / or the reflection. For example, in the case of a medical device as the object, only relevant parts of the object and / or the reflection are then imaged, while others that are, for example, irrelevant or disruptive to the procedure are displayed transparently. In the case of schematic detection and display, however, the entire object and / or its mirror remains visible in the form of outlines and thus shows a precise spatial orientation without being displayed three-dimensionally.This allows the image to be reduced to essential elements so that a larger focus on essential elements is not compromised or distracted.

[0055] In a particularly preferred embodiment of the system according to the present disclosure, it further comprises an OCT system (as described above) and a surgical microscopic system configured to capture a time-resolved image signal of the selected field of view of the sample. The surgical microscopic system comprises an image sensor configured to capture a time-resolved image signal of the selected field of view (region of interest - ROI) of a sample, and preferably an optics system that may include, for example, an objective lens and an eyepiece, but may also include further components, in particular further lenses, mirrors, beam splitters, and / or the like. The optics and the image sensor are preferably integrated, for example, as parts of a camera of a surgical microscope.According to this embodiment, the control unit is further designed and configured to determine video image data corresponding to the acquired time-resolved image signal. The time-resolved image signal is, in particular, a plurality of signals associated with surface elements of the sample, which are acquired sequentially or simultaneously for a specific grid of the sample surface, wherein the grid is determined by a scanning mechanism and / or the image sensor. The time-resolved image signal further has a clock frequency (refresh rate) determined by a scanning mechanism and / or the image sensor. From this image signal, the control unit generates video image data with a grid (resolution) and a refresh rate suitable for display on the display means.The control unit is further configured to display the video image data simultaneously or sequentially with the time-resolved OCT image and / or with the virtual surface on the display means.

[0056] A further aspect of the present disclosure relates to a method for visualizing OCT signals. This comprises the method step of receiving a time-resolved OCT signal from a selected field of view of a sample from an OCT system (as described above). The OCT signal comprises a plurality of tuples, each of which comprises or represents a volume element of the sample and a scattering intensity corresponding to the volume element. The method further comprises the step of determining a time-resolved OCT image with at least one object and with a virtual surface, wherein a reflection of the at least one object of the OCT image is determined on the virtual surface. Finally, in the method according to the present disclosure, the time-resolved OCT image with the at least one object and with the virtual surface is displayed on the display means.The method according to the present disclosure realizes the same advantages as the system according to the present disclosure and in this regard reference is made to the above explanations.

[0057] In a preferred embodiment of the method according to the present disclosure, it further comprises the step of determining the at least one object by segmenting the OCT signal or by segmenting the OCT image, preferably using (based on) object-specific information of the at least one object and / or a user input. Thus, the reflection of semantically segmented objects is advantageously generated on the virtual surface.

[0058] Further preferred implementations of the method according to the present disclosure correspond to further preferred embodiments of the system according to the present disclosure and realize the same advantages as the embodiments.

[0059] A further aspect of the present disclosure relates to a computer program comprising instructions which, when executed by a control unit as described above, preferably of a surgical microscope as described above, cause the system or surgical microscope as described above to carry out the inventive method as described above. The computer program preferably comprises instructions which, when executed by a control unit as described above, preferably of a surgical microscope as described above, cause the system or surgical microscope as described above to carry out the inventive method according to one of the preferred embodiments as described above.The computer program according to the invention is preferably stored in a volatile memory, for example a RAM element, or in a non-volatile storage medium, such as a CD-ROM, a flash memory or the like.

[0060] Further preferred embodiments of the invention emerge from the remaining features recited in the subclaims and the figures explained below. The various embodiments of the invention recited in this application can be advantageously combined with one another, unless otherwise stated in individual cases. Description of the characters

[0061] The invention is explained below in exemplary embodiments with reference to the accompanying drawings. They show: Fig. 1 is a schematic representation of a system for visualizing OCT signals according to a first embodiment; Fig. 2 is a schematic representation of a system for visualizing OCT signals according to a second embodiment; Fig. 3 a schematic representation of the visualization of an OCT image with at least one object and with a virtual surface; Fig. 4a-4d show a representation of the methodology for visualizing an OCT image with at least one object and with a virtual surface; and Fig. 5-7 different views of a visualization of an OCT image with at least one object and with a virtual surface; Fig. 8 shows a schematic flow diagram of a method according to one embodiment; Fig. 9 temporal development of a distance error; and Fig. 10 Results of a NASA TLX survey regarding various stress factors.

[0062] Fig. 1 shows a schematic representation of a system 100 for acquiring and visualizing OCT signals 19 according to a first embodiment.

[0063] The system 100 comprises an OCT system 10 with a broadband light source 11, for example, a superluminescence diode. The light from the light source 11 is directed into an interferometer having a beam splitter 14 and a movable mirror 15. In the beam splitter 14, the light is split into a sample beam 12 and a reference beam 13. The sample beam 12 is scanned across the sample 65 by means of a scanning mirror 16, scanning at least a portion of a selected field of view 66 of the sample 65. The reference beam 13 is directed onto the movable mirror 15 and reflected back from it to the beam splitter 14. The sample beam 12 interacts with the sample 65, in particular the volume of the sample 65, and is scattered back by the scanning mirror 16, which directs it to the beam splitter 14.There, the backscattered sample beam 12 and the reflected reference beam 13 are superimposed, with a path difference between the superimposed beams 12, 13 being adjusted by the movable mirror 15. The interference pattern 17 thus generated is detected by a detector 18, for example, a CCD detector or a CMOS detector.

[0064] The time-resolved OCT signal 19 thus acquired is transmitted from the detector 18 to the control unit 40. The control unit 40 is further connected to a user interface 50 for detecting a user input and to a device interface 55 for receiving a device parameter. The device interface 55 serves, for example, to receive a device parameter of a device connected to the device interface 55, for example, a surgical microscope system 20, another imaging system, or a medical instrument. Furthermore, the device parameter can originate from a tracking system (not shown) connected to the device interface 55 but characterize the medical instrument, for example, its spatial position.

[0065] The control unit 40 determines a time-resolved OCT image 31 with at least one object 70 and with a virtual surface 60 for the acquired time-resolved OCT signal 19, wherein a reflection 70' of the at least one object 70 of the OCT image is determined on the virtual surface 60, and transmits the time-resolved OCT image 31 with the at least one object 70 and with the virtual surface 60 to the display means 30 for display.

[0066] Fig. Figure 2 shows a schematic representation of a system 100 for acquiring and visualizing OCT signals 19 according to a second embodiment. The same components are designated by the same reference numerals as in Fig. 1 and a repeated description of these components is therefore omitted.

[0067] The System 100 of Fig. 2 differs from that of the Fig. 1 in that it further comprises a surgical microscopic system 20 with a camera 21. The camera 21 comprises an image sensor 22 and an optics 23 and is designed to capture the field of view 66 of the sample 65, so that a time-resolved image signal 27 is captured as a result. The camera 21 can be a mono camera or a stereo camera. The control unit 40 determines video image data 32 corresponding to the captured time-resolved image signal 27. A time-resolved OCT signal 19 is generated in the system 100 of the Fig. 2 in the same way and an OCT image 31 with the at least one object 70 and with the virtual surface 60 is determined therefrom, as with reference to Fig. 1. The video image data 32 are displayed together with the OCT images 31 simultaneously or sequentially on the display means 30, with switching between the OCT images 31 and the video image data 32 occurring, for example, based on a user input acquired via the user interface 50. The surgical microscopic system 20 further comprises a device for splitting the observation beam path 26, which splits the light beam sent from the field of view 66 of the sample 65 to the camera 21 and provides a partial beam to the eye 24 of an observer via an eyepiece 25.

[0068] Fig. 3 shows a schematic representation of the visualization of an OCT image 31 with at least one object 70 and a virtual surface 60. The OCT image 31 is determined, as described, from the OCT signal 19 acquired from the field of view 66 of the sample 65, wherein an object 70, in this case a medical instrument, is located in the vicinity of the sample 65 during signal acquisition. The OCT image 31 comprises an image of a surface 63 of the field of view 66 of the sample 65, an image of the medical instrument, and an image of an anatomical structure of the field of view 66 of the sample 65. The light of the broadband light source 11 of the OCT system 10 is directed onto the sample 65 by the scanning mirror 16 such that the light rays impinge in the field of view 66 almost parallel to one another and at a right angle to the sample surface.

[0069] The OCT image 31 further includes a reflection 70' of the at least one object 70 on the virtual surface 60. This reflection was calculated as a reflection of the medical instrument on the virtual surface 60. Furthermore, the virtual surface 60 is semi-transparent, so that the anatomical structure behind the virtual surface 60 remains visible.

[0070] Fig. 4a to 4d show a methodology for visualizing OCT signals performed by the system according to the invention. Fig. 4a shows the original OCT volume. A virtual surface 60, alternatively referred to as an intelligent virtual B-scan mirror (IVBM), is determined as an extension of a selected virtual B-scan. According to the present preferred embodiment, the virtual surface 60 fulfills the requirements that voxels integrated into the IVBM are highlighted in the volume rendering while simultaneously being rendered semi-transparent to preserve volume structures behind the IVBM, and that the selected B-scan cross-section acts as an intelligent mirror by being sensitive only to voxels connected to the medical instrument in the vicinity of the virtual surface 60. The following sections describe the required components as well as the construction and direct integration of the IVBM into a volume raymarching algorithm.

[0071] Since the virtual surface 60 integrates a mirror that is only sensitive to medical instruments, candidate mirror voxels must be identified in the OCT volume. To achieve the real-time processing rates required for 4D OCT visualizations, the instrument is first identified in a 2D projection image 33 of the OCT volume. This is shown in Fig. 4b. A 2D projection image 33 or an en-face view 33 is determined, which encodes the average intensity along a plurality of OCT-A scans of the OCT signal. Based on this 2D projection image 33, the at least one object 70 is determined by segmentation in the en-face view. For this purpose, the 2D projection image is forwarded to a U-Net-like convolutional neural network to generate a binary instrument map M toolSince the OCT signal 19 is completely blocked at the surface of the medical instrument, anatomical structures below the instrument are obscured, and a corresponding plurality of A-scans of the plurality of A-scans containing the at least one object 70 contains only instrument-related voxels. A position of the at least one object 70 in the depth direction in the plurality of A-scans in the form of a 3D position of the voxels associated with the instrument can therefore be determined based on the detected plurality of A-scans in the binary instrument map M tool as described below. Before the OCT volume and the binary instrument map M toolare forwarded to a suitable DVR algorithm, the OCT volume is further processed by applying a 3D median filter to reduce speckle noise in the OCT. With these components, the IVBM composition is realized entirely within the DVR algorithm, as described in the following section.

[0072] The connections are in Fig. 4c. If an observation beam r B 61, which corresponds to an observation direction, starting from an observation point, intersects with the virtual surface 60, as in Fig. As shown in Figure 4c, three components contribute to the resulting ray color. Firstly, each voxel P m along the observation beam r B 61, which is assigned to and forms the virtual surface 60. Furthermore, each voxel P v , which is along an observation ray r B61 is arranged behind the virtual surface 60 in the direction of observation. On the other hand, each specific voxel P t , which is assigned to the at least one object 70, on which a beam emanating from the observation point and along the observation beam r B 61 and reflected by the virtual surface 60 ray r M 62. This is represented here by the mirror beam r m 62 is shown, which is reflected by the virtual surface 60 and intersects with the medical instrument.

[0073] The present virtual surface 60 is formed according to this preferred embodiment as any plane, either manually or automatically selected, which can thus be defined by the general equation: xnx+yny+znz+d=0 where n = (n x , n y , n z) indicates the normal of the plane. Furthermore, a distance threshold d IVBM which specifies the thickness of the virtual surface 60. During volume raymarching, a voxel P m with the position (P x ,P y ,P z ) is integrated into the virtual surface 60 if the following condition is met: ||xnx+yny+znz+d|nx2+ny2+nz2| <dIVBM

[0074] To visualize the reflections of the medical instrument, from each point P m , which results from equation (2), a mirror beam r reflected from the virtual surface 60 m 62. The direction of the mirror beam is determined by: rm=rB−2∗dot(rB,n)∗n

[0075] During volume raymarching along r m 62, the binary instrument card M described above tool and an intensity threshold t tool= 0.25 (empirically determined to reject OCT speckle noise) is used to determine whether the sampled point is a mirror candidate: Mtool(Pt)=1∩I(Pt)>ttool where I indicates an intensity of a voxel. In order to further preferably only mirror the medical instrument when it is located in a defined proximity to the virtual surface 60, the number of scanned steps n along r m 62. In case of an intersection with an instrument, as determined in equation (4), the volume raymarching is terminated and the reflection 70' of the medical instrument in the virtual surface 60 at P m with a color component C Pt Preferably, the distance along r m 62 between P m in the virtual area 60 and P tEncoded on the medical instrument using a perceptually linear color map. Preferably, an L*a*b* color space with the following characteristics is used: CL∗a∗b∗(I,δ*)=γ(I)∗(δ*∗C0+(1−δ*)∗C1) where δ* = (i m ∗ 0.7) / n steps is a distance predicate that determines the step index i m ∈ [0,n steps ] along r m 62 when reaching the medical instrument at P t taken into account. γ(I) is a scaling factor which is also presented in “Layer-aware ioct volume rendering for retinal surgery.”, Weiss, J., Eck, U., Nasseri, MA, Maier, M., Eslami, A., Navab, N.: In: “Eurographics Workshop on Visual Computing for Biology and Medicine. The Eurographics Association”, Kozlfkova, B., Linsen, L., Vazquez, PP, Lawonn, K., Raidou, RG (eds.) (2019). By choosing C0 = (I(P t ), -1.5, 1) and C1 = (I(P t), -1.0, -1.0), a color interpolation is achieved between a blue hue when the medical instrument is farther away from the IVBM and a green hue when it is close to the virtual surface 60. The RGB component of the instrument's reflection 70' is then obtained with: CPt=[RGB(CL∗a∗b∗(I(Pt),δ*(p)))] where RGB(C) is a conversion of L*a* b* into the RGB color space. In addition, the opacity of the reflection 70' is determined with increasing distance of the medical instrument from the virtual surface 60 with σ(Pt)=1.0−(stepmnsteps)2 reduced.

[0076] The overall appearance of a voxel at P m , which integrates the reflection 70' and at the same time amplifies the natural intensities in the virtual surface 60, is finally defined by: C(Pm)=[μ∗I(Pm)∗(1−σ(Pt))+C(Pt)∗σ(Pt),α(μ∗I(Pm))]

[0077] In practice, µ is a dynamically modifiable parameter for enhancing the volume intensities of the virtual B-scan, and α(I) is a conventional piecewise linear opacity function. During volume raymarching, alpha blending is used to integrate the virtual surface 60 with the remaining volume structures according to equation (2). In general, voxels P v that are not included in the IVBM can be rendered using any direct volume rendering convention. In our experiments and for the visualizations in Fig. Figures 4 to 7 use the classic Phong shading, which was previously used for OCT volume rendering. It provides structural surface details in the visualizations while visually highlighting the virtual surface 60 in the final rendering.

[0078] The result of the methodology is in Fig. 4d. It is clearly visible that the medical instrument is determined and displayed as a reflection 70' in the virtual surface 60. Nevertheless, volume structures behind the virtual surface 60 are still recognizable.

[0079] Fig. 5 shows another view of the OCT image 31. There, a visual highlighting of volume intensities integrated into the virtual surface 60 can be seen on the virtual surface 60 of the OCT image 31. In other words, voxels integrated into the virtual surface 60, for example, in volume rendering, are highlighted. Furthermore, target structures aligned with the virtual surface 60 are enhanced. These target structures preferably include, among others, anatomical target structures that are displayed with greater illumination.

[0080] As in the Fig. 6 and Fig. As shown in Figure 7, the IVBM has the advantage of providing perceptual cues by visualizing the intersection of the instrument with the virtual surface 60 as the tip of the instrument meets the tip of the mirrored instrument 70'. It is also capable of visualizing instrument structures that are obscured in direct view or located behind the IVBM and enhances target structures aligned with the IVBM.

[0081] Fig. 8 shows a schematic flow diagram of a method according to one embodiment. The method comprises a first step S100 of receiving a time-resolved OCT signal 19 of a selected field of view 66 of a sample 65 from an OCT system 10. The OCT signal 19 comprises a plurality of tuples, each representing a volume element of the sample 65 and a scattering intensity corresponding to the volume element. The method further comprises the second step S200 of determining S200 a time-resolved OCT image 31 with at least one object 70 and with a virtual surface 60, wherein a reflection 70' of the at least one object 70 of the OCT image is determined on the virtual surface 60. Finally, in a third step S300, the time-resolved OCT image 31 is displayed on the display means 30.

[0082] The inventors further evaluated the feasibility of using IVBMs for live visualization of 4D OCT data by implementing the proposed concept into a visualization framework. 2D projection image generation and IVBM-integrated volume raymarching were implemented using OpenGL 4.6 and tested on an NVidia RTX 3090Ti GPU. An instrument segmentation network was trained on a custom dataset consisting of 3356 2D projection images generated from OCT volumes with a resolution of 391 x 391 x 644 voxels, containing a synthetic eye model and surgical forceps, and incorporating random rotations and reflections for data augmentation. PyTorch 1.13 and TensorRT 8.4 were used for model training and optimization. The data was labeled by two biomedical engineers.

[0083] The average total processing and rendering time based on 20 test volumes with the same resolution was 44.3 (±3.1) ms (filtering: 8.1 (±1.2) ms, generation of projection images and segmentation of the instruments: 5.7 (±2.6) ms, rendering: 30.5 (±0.6) ms). These benchmarks were run with n steps = 120 mirror sample steps. To demonstrate the 4D interactions, the method was integrated into a 4D SS-OCT system provided by a medical device company. OCT volumes were acquired with a resolution of 391 x 391 x 644 at volume update rates of 10 Hz.

[0084] To determine whether an IVBM can support users in performing targeted instrument maneuvers under 4D-OCT guidance, a user study was conducted in which participants were asked to move the tip of a surgical instrument to defined target locations. To achieve continuous, accurate, and efficient data acquisition during the study, a virtual environment with simulated 4D-OCT was used, using the method proposed in Sommersperger, M., Martin-Gomez, A., Mach, K., Gehlbach, PL, Nasseri, MA, lordachita, I., Navab, N.: Surgical scene generation and adversarial networks for physics-based ioct synthesis. Biomedical Optics Express 13(4), 2414-2430 (2022).Additionally, a 3D haptic input device (3D Systems Geomagic Touch) was integrated to navigate the virtual surgical instruments, applying a motion scaling of 4:1 to reduce the influence of the users' individual manual tremor. Small targets were generated on the retina, and the IVBM was automatically positioned at the target locations. To measure the effectiveness of the IVBM in interacting from unusual perspectives, the virtual scene was rendered from a fixed view approximately orthogonal to the A-scan direction. The renderings were displayed in a VR headset that utilized stereo rendering. Users were instructed to navigate the instrument tip to the target location and press a button once they were satisfied with the positioning. Participants included a total of 15 biomedical experts (12 male, 3 female) familiar with ophthalmology and OCT.The study was conducted in accordance with the Declaration of Helsinki, study data were anonymized, and vision tests were conducted prior to the study to ensure participants' healthy vision. After participants became familiar with interacting in the virtual environment, they performed eight trials with IVBM enabled. For ablation, the same number of trials were performed without IVBM, using the method for 4D OCT-DVR with Phong shading proposed in Viehland, C., Keller, B., Carrasco-Zevallos, OM, Nankivil, D., Shen, L., Mangalesh, S., Kuo, AN, Toth, CA, Izatt, JA, et al.: Enhanced volumetric visualization for real time 4d intraoperative ophthalmic swept-source oct. Biomedical optics express 7(5), 1815-1829 (2016) as a basis. Positioning accuracy and distance to the target were measured over the course of the trials.The results show that users reached the target with an average error of 70 µm (±40 µm) when the IVBM was activated, while an average error of 135 µm (±128 µm) was measured with baseline data playback, indicating statistically significant differences between the distributions (p < 0.002 based on a Kruskal-Wallis test after determining unequal variances). Furthermore, the distance between the instrument tip and the target was analyzed in relation to the course of the trial. Fig. Figure 9 shows that with the IVBM activated, the range error deviation decreased continuously, especially in the last quarter of the test, resulting in more accurate and precise target acquisition. The results of the NASA TLX survey in Fig.10, conducted after the study, showed improvements in all categories when the IVBM was activated. However, statistical significance was only found in the categories of performance (p < 0.001) and physical exertion (p < 0.02) based on an equal variance ANOVA test.

[0085] When users were provided with an IVBM, statistically significant improvements in aiming error were observed. The results of the subjective assessment of task load were also overall better when the IVBM was activated, although statistical significance was not achieved in categories such as mental demand or frustration. Interestingly, higher aiming accuracy was still achieved with the IVBM, even when users reported higher effort or rated their performance as worse compared to baseline. Abstracting the mirrored view to incorporate only important key points of the instrument could further reduce the mental effort for users. One advantage of the IVBM is the direct in-situ visualization, which also highlights anatomical target structures, as the user does not have to avert their gaze from the surgical field to perceive the mirrored view. List of reference symbols 10 OCT system 11 broadband light source 12 Sample beam 13 Reference beam 14 beam splitters (interferometers) 15 movable mirrors (interferometers) 16 Scanning mechanism (scanning mirror) 17 interference patterns 18 detector 19 time-resolved OCT signal 20 surgical microscopic system 21 Camera 22 image sensor 23 Optics 24 Eye of a Beholder 25 eyepiece 26 Device for splitting the observation beam path 27 time-resolved image signal 30 display devices 31 OCT image 32 video image data 33 2D projection image / en face view 40 Control unit 50 User interface 55 Device interface 60 virtual area / IVBM 61 Observation beam r B 62 reflected beam / mirror beam r M 63 area 65 Sample 66 field of view 70 Object / medical instrument 70' reflection of the object / medical instrument

Claims

[1] System (100) for visualizing OCT signals, comprising: a display means (30) designed for the time-resolved display of image data; and a control unit (40) which is arranged to to receive a time-resolved OCT signal (19) of a selected field of view (66) of a sample (65) from an OCT system (10); to determine a time-resolved OCT image (31) with at least one object (70) and with a virtual surface (60) based on the OCT signal (19), wherein a reflection (70') of the at least one object (70) of the OCT image (31) is determined on the virtual surface (60); and to display the time-resolved OCT image (31) on the display means (30). [2] The system (100) of claim 1, wherein the reflection (70') of the at least one object (70) on the virtual surface (60) is determined when a ray (62) originating from an observation point and reflected from the virtual surface (60) intersects the at least one object (70). [3] System (100) according to claim 1 or 2, wherein the virtual surface (60) is formed by a plurality of voxels and the time-resolved OCT image (31) in the virtual surface (60) is determined based on an OCT signal (19) associated with voxels forming the virtual surface (60). [4] System (100) according to one of the preceding claims, wherein an OCT signal (19) associated with voxels forming the virtual surface (60) is displayed in an amplified manner in the time-resolved OCT image (31). [5] System (100) according to one of the preceding claims, wherein the time-resolved OCT image (31) on or in the virtual surface (60) is further determined based on an OCT signal associated with voxels arranged behind the virtual surface (60) in an observation direction. [6] System (100) according to one of the preceding claims, wherein an OCT signal associated with voxels arranged behind and / or in front of the virtual surface (60) in an observation direction is displayed in an attenuated manner in the OCT image (31). [7] System (100) according to one of the preceding claims, wherein the at least one object (70) comprises a medical instrument (70) and / or an anatomical structure. [8] System (100) according to one of the preceding claims, wherein the control unit (40) is configured to determine the at least one object (70) by segmenting the OCT signal (19) or the OCT image (31). [9] System (100) according to one of the preceding claims, wherein the OCT signal (19) comprises a plurality of A-scans and the control unit (40) is arranged to to determine an en face view (33) based on the large number of A-scans, to determine the at least one object (70) by segmentation in the en face view (33), to determine a plurality of A-scans of the plurality of A-scans containing the at least one object (70), and to determine a position of the at least one object (70) in the depth direction in the plurality of A-scans. [10] System (100) according to one of the preceding claims, further comprising an interface configured to detect a user input, wherein the control unit (40) is configured to segment a plurality of objects (70) in the OCT signal (19) or the OCT image (31) and to select the at least one object (70) therefrom based on the user input. [11] System (100) according to one of the preceding claims, wherein the control unit (40) is configured to determine a color coding of the reflection (70') of the at least one object (70) based on a distance between the at least one object (70) and the virtual surface (60) in the time-resolved OCT image (31). [12] System (100) according to one of the preceding claims, wherein the control unit (40) is configured to vary a position and / or orientation of the observation point, the observation direction and / or the virtual surface (60). [13] System (100) according to claim 12, wherein the system (100) further comprises an interface configured to detect a user input and the control unit (40) is configured to adjust the position and / or orientation of the observation point, the observation direction and / or the virtual surface (60) based on the user input and / or wherein the system (100) further comprises a device interface (55) configured to detect a device parameter and the control unit (40) is configured to adjust the position and / or orientation of the observation point, the observation direction and / or the virtual surface (60) based on the device parameter. [14] System (100) according to claim 12 or 13, wherein the control unit (40) is arranged to to identify a target structure based on anatomical information, and to adjust the position and / or orientation of the observation point, the observation direction and / or the virtual surface (60) based on the determined target structure. [15] System (100) according to any one of the preceding claims, wherein the virtual surface (60) comprises a concave or convex surface. [16] System (100) according to one of the preceding claims, wherein the at least one object (70) and / or the reflection (70') of the at least one object (70) is determined pointwise and / or schematically in the time-resolved OCT image (31). [17] System (100) according to one of the preceding claims, further comprising: an OCT system (10) a surgical microscopic system (20) designed to acquire a time-resolved image signal (27) of the selected field of view (66) of the sample (65), wherein the control unit (40) is further configured to determine corresponding video image data (32) on the basis of the acquired time-resolved image signal (27), and to display the video image data (32) simultaneously or sequentially with the time-resolved OCT image (31) and / or with the virtual surface (60) on the display means (30). [18] Method for visualizing OCT signals (19), comprising the method steps: Receiving (S100) a time-resolved OCT signal (19) of a selected field of view (66) of a sample (65) from an OCT system (10), wherein the OCT signal (19) comprises a plurality of tuples, each representing a volume element of the sample (65) and a scattering intensity corresponding to the volume element (651); Determining (S200) a time-resolved OCT image (31) with at least one object (70) and with a virtual surface (60), wherein a reflection (70') of the at least one object (70) of the OCT image (31) is determined on the virtual surface (60); and Displaying (S300) the time-resolved OCT image (31) on the display means (30).

Images