Surgical microscope system and system, method and computer program for surgical microscope system

Abstract

Examples relate to surgical microscope systems (100) and systems (110), methods, and computer systems for surgical microscope systems. The system (110) is configured to obtain first imaging sensor data of a view on a surgical site from a first optical imaging sensor (122) of a microscope (120) of a surgical microscope system. The system is configured to obtain second imaging sensor data of a view on the surgical site from a second sensor (124) of the microscope. The system is configured to determine an extent of one or more anatomical features of the surgical site based on the second imaging sensor data. The system is configured to apply spatially varying noise filtering to the first imaging sensor data based on a range of one or more anatomical features of the surgical site.

Description

Technical Field

[0001] Examples relate to surgical microscope systems, and systems, methods, and computer systems for surgical microscope systems. Background Art

[0002] Digital microscopes, and in particular digital surgical microscopes, typically have multiple imaging modes - such as a reflectance imaging mode, in which light reflected from the sample being imaged is used to generate a digital view of the sample; and a fluorescence imaging mode, in which fluorescence emissions from fluorophores applied to the sample are used to generate a digital view of the sample.

[0003] Compared to the reflection used for reflectance imaging, fluorescence emission has significantly lower light intensity, so the corresponding optical imaging sensors usually operate at higher sensitivity. As a result, fluorescence images tend to exhibit more noise. Although there are various noise filtering methods, noise filtering of fluorescence images remains a challenging topic and the filtered images are often of suboptimal quality. In general, existing noise filtering methods are generally applied to the entire image.

[0004] More effective filtering of noise can improve image quality, especially increasing contrast. Summary of the invention

[0005] There may be a need for improved concepts for noise filtering of microscope images of lower intensity light sources.

[0006] This expectation is solved by the subject-matter of the independent claims.

[0007] Various examples of the present disclosure are based on the discovery that different types of anatomical features can appear very differently in imaging sensor data generated by a digital surgical microscope. For example, in a reflectance image of the brain, (blood) vessels can appear very clearly as red or purple features, while other types of tissue may appear as a lower intensity flesh-colored background. Similarly, in a fluorescent image of the brain, blood vessels containing fluorophores appear as relatively bright portions of the fluorescent image, while blood vessels or other types of tissue that do not contain fluorophores remain dark. In order to improve the noise reduction effect, these different portions of the corresponding images can be processed differently so that the noise filtering applied to the imaging sensor data is spatially varying. In order to improve the accuracy of the spatially varying noise filtering, the range of the corresponding anatomical features that are processed differently is determined using imaging sensor data that clearly shows the range and / or type of the corresponding features, and the actual spatially varying noise filter is applied to another set of imaging sensor data, where the determination of the range may be less precise or more difficult.

[0008] Various examples of the present disclosure relate to a system for a surgical microscope system. The system includes one or more processors and one or more storage devices. The system is configured to obtain first imaging sensor data of a view on a surgical site from a first optical imaging sensor of a microscope of the surgical microscope system. The system is configured to obtain second imaging sensor data of a view on a surgical site from a second sensor of the microscope. The system is configured to determine the range of one or more anatomical features of the surgical site based on the second imaging sensor data. The system is configured to apply a spatially varying noise filter to the first imaging sensor data based on the range of one or more anatomical features of the surgical site. By changing the spatially varying noise filter based on the range of one or more anatomical features, the noise filter can be adapted to the anatomical features shown in the first and second imaging sensor data. By using the second imaging sensor data to determine the range of one or more anatomical features, the accuracy of the range determination can be increased.

[0009] For example, the first imaging sensor data may be based on fluorescence imaging. In this case, spatially varying noise filtering may be particularly effective because a more aggressive noise filter may be used in portions of the imaging sensor data that do not exhibit fluorescence emission, while a specialized noise filter may be used in portions of the imaging sensor data that exhibit fluorescence emission. In addition, fluorescence imaging sensor data is typically low in contrast, so if another set of imaging sensor data (i.e., using the second imaging sensor data) is used, the determination of the extent of one or more anatomical features may be improved.

[0010] For example, the second imaging sensor data may be based on reflectance imaging. Alternatively, or in addition, other imaging techniques may be used, such as imaging spectroscopy, including multispectral, hyperspectral and derivatives, reflectance and / or fluorescence, Raman imaging and derivatives, laser speckle imaging and derivatives, confocal imaging and derivatives, optical property imaging, i.e., μ a (absorption coefficient) and μ s (scattering coefficients), ultrasound imaging, photoacoustic imaging and derivatives, 3D surface scanning, kinetic mapping imaging (e.g., indocyanine green (ICG) bolus kinetics), functional imaging by any modality, preoperative or intraoperative, preoperative or real-time, or anatomical estimation imaging derived from comparative tissue imaging in conjunction with an anatomical database. Various types of imaging may be appropriate to determine the extent of an anatomical feature depending on the type of anatomical feature and the type of imaging modality available.

[0011] In general, not all anatomical features are relevant with respect to spatially varying filtering. For example, only some blood vessels may contain fluorophores, while other blood vessels and other types of tissue may not light up in fluorescence imaging. Therefore, a subset of anatomical features (i.e., anatomical features of interest) can be selected from the one or more anatomical features determined. The system can be configured to determine at least one feature of interest among the one or more anatomical features of the surgical site and apply spatially varying noise filtering based on the range of the at least one feature of interest. For example, as described above, the one or more anatomical features can be one or more blood vessels, and the at least one feature of interest can be at least one blood vessel that emits fluorescent emissions.

[0012] Whether an anatomical feature is an anatomical feature of interest can be determined based on different criteria, for example, based on a classification of the respective anatomical feature. In some examples, the first imaging sensor data can be used to classify between anatomical features of interest and anatomical features of no interest. For example, the system can be configured to determine at least one feature of interest among the one or more anatomical features based on the first imaging sensor data.

[0013] In some cases, it may not be possible to distinguish whether an anatomical feature is an anatomical feature of interest in each frame of the first imaging sensor data. Therefore, multiple frames of the first imaging sensor data may be processed to determine whether the anatomical feature is of interest. For example, the system may be configured to determine at least one feature of interest among the one or more anatomical features based on multiple frames of the first imaging sensor data covering a predefined time interval or two or more predefined time points.

[0014] In many cases, it is sufficient to apply two different types of noise filtering - a first type for anatomical features (of interest) and a second type for background (e.g., tissue of little or no interest in surgery). For example, the system can be configured to subdivide the first imaging sensor data into a first portion and a second portion, the first portion being based on the extent of at least a subset of one or more anatomical features of the surgical site. The system can be configured to apply a first noise filter to the first portion and a different second noise filter to the second portion.

[0015] For example, as described above, the first imaging sensor data may be subdivided into a first portion that includes anatomical features (of interest) and a second portion that includes the remainder (such as anatomical features not of interest and other types of tissue). For example, the system may be configured to subdivide the first imaging sensor data such that the first portion includes the extent of at least a subset of one or more anatomical features of the surgical site, while the second portion includes the remainder of the first imaging sensor data.

[0016] In fluorescence imaging, some assumptions may be made about the expected behavior of anatomical features containing fluorophores. One assumption is that the intensity of the fluorescence emission primarily follows a known intensity pattern. Accordingly, a first noise filter may be configured to apply a predefined intensity pattern to a first portion of the first imaging sensor data. In particular, the intensity pattern of the fluorescence emission is mostly homogeneous within the vessel. Accordingly, the predefined intensity pattern may be configured to apply a uniform intensity distribution to the first portion or a coherent subportion of the first portion. However, some gradual changes may be displayed, such as a gradient along the longitudinal axis of the corresponding feature. For example, the predefined intensity pattern is configured to apply a uniform intensity distribution along the horizontal axis of the coherent subportion of the first portion, and to apply an intensity gradient along the longitudinal axis of the coherent subportion of the first portion.

[0017] The proposed concept is particularly suitable for noise filtering of blood vessels. Blood vessels, especially blood vessels of the brain, can be tree-like structures with many branches. The distribution of fluorophores within the blood vessels may be affected by the branches of the blood vessels. Accordingly, the blood vessels can be subdivided into smaller sub-portions, and noise filtering can be applied to each smaller sub-portion separately. For example, the system can be configured to subdivide the first part into multiple coherent sub-portions based on the range of at least a subset of one or more anatomical features, and apply the first noise filter to the coherent sub-portions separately. For example, the one or more anatomical features can be one or more blood vessels. The system can be configured to subdivide the first part into multiple related sub-portions based on the branch points of one or more blood vessels. This can improve the quality of noise filtering because subtle gradients between sub-portions can be modeled more accurately.

[0018] For background or less important anatomical features, coarser and / or more aggressive noise filtering may be applied.For example, the second noise filter may be configured to apply a spatial low pass filter to the second portion of the first imaging sensor data.

[0019] In general, at least two techniques may be used to determine the extent of the surgical site - a machine learning based approach and a color / shape based approach. For example, the system may be configured to determine the extent of one or more anatomical features of the surgical site using a machine learning model trained to perform image segmentation and / or object detection. Alternatively, or in addition, the system may be configured to determine the extent of one or more anatomical features of the surgical site based on at least one of a characteristic color spectrum and a characteristic shape of the one or more anatomical features of the surgical site. Both techniques have advantages and disadvantages, for example, in terms of computational effort, implementation complexity, and / or traceability.

[0020] The result of the noise filtering can be displayed via a display device of the surgical microscope system. For example, the system can be configured to generate a display signal for a display device of the surgical microscope system based on at least the filtered first imaging sensor data. For example, the system can be configured to generate a composite digital view of the surgical site based on the filtered first imaging sensor data and based on the second imaging sensor data, and to generate a display signal based on the composite digital view. In other words, the first and second imaging sensor data can be combined into a single image and displayed together on the display device.

[0021] Various examples of the present disclosure relate to corresponding methods for surgical microscope systems. The method includes obtaining first imaging sensor data of a view on a surgical site from a first optical imaging sensor of a microscope of the surgical microscope system. The method includes obtaining second imaging sensor data of a view on a surgical site from a second sensor of the microscope. The method includes determining the extent of one or more anatomical features of the surgical site based on the second imaging sensor data. The method includes applying a spatially varying noise filter to the first imaging sensor data based on the extent of the one or more anatomical features of the surgical site.

[0022] Various examples of the present disclosure relate to a corresponding computer program with a program code for performing the above-mentioned method when the computer program is executed on a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Some examples of apparatus and / or methods will be described below, by way of example only, and with reference to the accompanying drawings, in which

[0024] Figure 1a A schematic diagram showing an example of a system for a surgical microscope system;

[0025] Figure 1b A schematic diagram showing an example of a surgical microscope system;

[0026] Figure 1c A schematic diagram showing an example of a noisy fluorescence image of a surgical site;

[0027] Figure 1d A schematic diagram showing an example of a reflected image of a surgical site;

[0028] Figure 2 A flow chart showing an example of a method for a surgical microscope system;

[0029] Figure 3 A schematic diagram showing application of noise filtering to a noisy fluorescence image; and

[0030] Figure 4 A schematic diagram of an example of a system including a microscope and a computer system is shown. DETAILED DESCRIPTION

[0031] Various examples will now be described more fully with reference to the accompanying drawings, in which some examples are shown. In the drawings, the thickness of lines, layers and / or regions may be exaggerated for clarity.

[0032] Figure 1a A schematic diagram of an example of a system 110 for use with a surgical microscope system 110 is shown. The surgical microscope system 110 includes a microscope 120 that is a digital microscope, i.e., it includes at least one optical imaging sensor 122; 124, which is coupled to the system 110. In general, a microscope, such as the microscope 120, is an optical instrument suitable for examining objects that are too small to be examined by the human eye (alone). For example, a microscope can provide optical magnification of a sample. In modern microscopes, optical magnification is typically provided by a camera or imaging sensor, such as the first optical imaging sensor 122 and / or the second sensor 124 of the microscope 120. In Figure 1a , the microscope 120 is shown as having two optical imaging sensors 122; 124. However, the second sensor 124 is not necessarily an optical imaging sensor, but can be any type of sensor capable of providing an image representation of the sample 10. The microscope 120 may also include one or more optical magnification components for magnifying the view on the sample, such as an objective (i.e., a lens).

[0033] In addition to the optical components that are part of the microscope 120, the surgical microscope system 100 also includes a system 110, which is a computer system. The system 110 includes one or more processors 114 and one or more storage devices 116. Optionally, the system also includes one or more interfaces 112. The one or more processors 114 are coupled to the one or more storage devices 116 and optionally the one or more interfaces 112. In general, the functionality of the system is provided by the one or more processors 114 in conjunction with the one or more interfaces 112 (for exchanging information, for example, with the first optical imaging sensor 122, with the second sensor 124, and / or with a display device, such as a visual display 130a or an auxiliary display 130b) and / or the one or more storage devices 116 (for storing and / or retrieving information).

[0034] The system 110 is configured to obtain first imaging sensor data of a view on the surgical site 10 from a first optical imaging sensor 122 of a microscope 120 of the surgical microscope system 100. The system 110 is configured to obtain second imaging sensor data of a view on the surgical site 10 from a second sensor 124 of the microscope 120. The system 110 is configured to determine the extent of one or more anatomical features of the surgical site based on the second imaging sensor data. The system 110 is configured to apply a spatially varying noise filter to the first imaging sensor data based on the extent of the one or more anatomical features of the surgical site. It is apparent that the system 110 is a system for processing imaging sensor data in the surgical microscope system 100 and / or for controlling the microscope 120 and / or other components of the surgical microscope system 100.

[0035] In general, a microscope system, such as surgical microscope system 100, is a system that includes a microscope 120 and additional components that operate with the microscope, such as system 110 (which is a computer system suitable for controlling the surgical microscope system and, for example, processing imaging sensor data of the microscope), as well as additional sensors, displays, etc.

[0036] There are various different types of microscopes. If the microscope is used in the medical or biological fields, the object 10 observed through the microscope may be a sample of organic tissue, for example, arranged in a culture dish or present in a part of the patient's body. In the present example, the microscope 120 is a microscope of a surgical microscope system, i.e. a microscope used in a surgical procedure (such as a neurosurgery procedure, a tumor surgery procedure) or a tumor surgery. Accordingly, the object observed through the microscope can be a sample of the patient's organic tissue, in particular, it can be a surgical site operated by a surgeon during the surgical procedure. Below, in the neurosurgery procedure, it is assumed that the object 10 to be imaged, i.e. the surgical site, is the surgical site of the brain. However, the proposed concept is also applicable to other types of surgical operations, such as cardiac surgery or ophthalmology.

[0037] Figure 1b A schematic diagram of an example of a surgical microscope system 100 is shown including system 110 and microscope 120 having a first optical imaging sensor 122 and a second optical imaging sensor 124. Figure 1bThe surgical microscope system 100 shown includes a number of optional components, such as a base unit 105 (including the system 110) and a (rolling) stand, a visual display 130a disposed on the microscope 120, an auxiliary display 130b disposed on the base unit 105, and a (robotic or manual) arm 140 that holds the microscope 120 in place and is coupled to the base unit 105 and the microscope 120. In general, these optional and non-optional components can be coupled to the system 110, and the system 110 can be configured to control and / or interact with the various components.

[0038] The proposed concept is based on processing two types of imaging sensor data - first imaging sensor data, which is imaging sensor data to be subjected to spatially varying noise filtering, and second imaging sensor data, which is imaging sensor data used to determine the extent of one or more anatomical features. Figure 1c and 1d Two examples of such imaging sensor data are shown. Figure 1c Schematic diagram showing an example of a noisy fluorescence image of a surgical site. In other words, the first imaging sensor data may be based on fluorescence imaging. Accordingly, the first optical imaging sensor may be an optical imaging sensor for performing fluorescence imaging. Alternatively, the first imaging sensor data may be based on other types of imaging based on sensing low light intensities. Figure 1d A schematic diagram showing an example of a corresponding reflected image of a surgical site. In other words, the second imaging sensor data can be based on reflectance imaging. For example, the second imaging sensor data can be based on hyperspectral reflectance imaging. Accordingly, the second sensor can be an optical imaging sensor for performing reflectance imaging (e.g., hyperspectral reflectance imaging). For example, the first and second imaging sensor data can have the same field of view, or the fields of view of the two sets of imaging sensor data can have a known spatial relationship. Figure 1d In the figure, different anatomical features, two blood vessels 14; 16 are clearly distinguishable. Between the anatomical features 14; 16, tissue 12a; 12b; 12c is shown, which is considered to be the "background" behind the anatomical features of interest. From the comparison of the figures, it is obvious that the blood vessels 14; 16 have different colors (visualized by two different line styles) due to the different oxygenation levels of the blood. Similarly, as Figure 1c As shown, the fluorescence emission from the two blood vessels also has different intensities due to the concentration of the fluorophore in the two blood vessels. Within the blood vessels, the intensity of the fluorescence emission can be considered to be homogeneous, or at least homogeneous along the horizontal axis, and has a gradient along the vertical axis.

[0039] In the above example, the second imaging sensor data is based on reflectance imaging, such as hyperspectral reflectance imaging. However, different types of sensor imaging sensor data may be used. In this case, the term "second imaging sensor data" means that the second imaging sensor data is obtained in the form of an image, i.e. as an image. In general, the second imaging sensor data may be an image obtained from an optical imaging sensor, or an image derived from other types of optical or non-optical sensor data, or a computer-generated image derived from a pre-operative scan or a database of anatomical features. For example, the second imaging sensor data may be based on or include at least one of: imaging spectroscopy (including multispectral, hyperspectral and derivatives, reflectance and / or fluorescence), Raman imaging (and derivatives), laser speckle imaging (and derivatives), confocal imaging (and derivatives), optical property images (particularly representing μ a (absorption coefficient) and / or μ s (scattering coefficient) optical property image or its derivative), ultrasound imaging, photoacoustic imaging (and derivatives), 3D surface scanning (e.g., using a depth sensor), dynamic mapping imaging (e.g., ICG bolus dynamics), functional images captured in any manner (preoperative, intraoperative, preoperative or real-time), and anatomical estimation images derived from comparative tissue imaging in conjunction with an anatomical database (e.g., showing where anatomical or functional tissue regions are located within the surgical site). Accordingly, the second sensor can be one of an optical imaging sensor (such as a hyperspectral optical imaging sensor, a multispectral optical imaging sensor, an optical imaging sensor for fluorescence imaging (for dynamic imaging and / or imaging spectroscopy), a spectral imaging sensor, a Raman spectral imaging sensor, an optical imaging sensor for laser speckle imaging, a confocal optical imaging sensor, or an optical imaging sensor for determining optical properties), an ultrasound sensor, a photoacoustic imaging sensor, and a depth sensor.

[0040] The second imaging sensor data is now used to determine the extent of one or more anatomical features of the surgical site. To determine the extent of the one or more anatomical features, the one or more anatomical features may be determined (e.g., detected, identified) in the second imaging sensor data. For example, the second imaging sensor data may be analyzed to determine and distinguish anatomical features in the imaging sensor data, such as blood vessels, tumors, branches, portions of tissue, etc. Generally speaking, two types of techniques may be used to perform the determination of extent.

[0041] In the first technique, a color-based and / or shape-based approach may be used. For example, the system may be configured to determine the extent of one or more anatomical features of the surgical site based on at least one of a characteristic spectrum and a characteristic shape of the one or more anatomical features of the surgical site. For example, each anatomical feature (of interest) may have a known characteristic spectrum and / or a characteristic shape. For example, depending on the oxygenation of the blood, the color of the blood vessel may be part of a known spectrum characterized by the blood vessel, ranging from bright red (arterial blood) to bluish purple (venous blood). For example, a blood vessel conveying arterial blood and a blood vessel conveying venous blood may consider separate characteristic spectrums. In addition, blood vessels typically have a characteristic shape with identifiable sharp edges in the second imaging sensor data. The system may be configured to identify a portion (e.g., a pixel) of the second imaging sensor data, which has a color that is part of a characteristic spectrum of the anatomical feature (of interest) and / or is part of a structure having a characteristic shape, and determine the extent of one or more anatomical features based on the portion (e.g., a pixel) of the second imaging sensor data having a color that is part of a characteristic spectrum of the anatomical feature (of interest) and / or is part of a structure having a characteristic shape.

[0042] Alternatively (or additionally), as a second technique, machine learning can be used to determine the extent of one or more anatomical features. To this end, one or both of the following machine learning-based techniques - image segmentation and object detection - can be used. In other words, the system can be configured to determine the extent of one or more anatomical features of the surgical site using a machine learning model trained to perform image segmentation and / or object detection. The system can be configured to perform image segmentation and / or object detection to determine the presence and / or location of one or more anatomical features in the imaging sensor data. In object detection, the location of one or more predefined objects (i.e., the objects on which the corresponding machine learning model is trained) in the imaging sensor data is output by the machine learning model, as well as the classification of the object (if the machine learning model is trained to detect multiple different types of objects). Typically, the location of one or more predefined objects is provided as a bounding box, i.e., a set of locations that form a rectangular shape surrounding each detected object. In image segmentation, the location of a feature (i.e., a portion of the second imaging sensor data having similar attributes, e.g., belonging to the same object) is output by the machine learning model. Typically, the location of the feature is provided as a pixel mask, i.e., the location of the pixel belonging to the feature is output according to each feature.

[0043] For object detection and image segmentation, machine learning models trained to perform the corresponding tasks are used. For example, in order to train a machine learning model trained to perform object detection, multiple samples of imaging sensor data can be provided as training input samples, and a corresponding list of bounding box coordinates can be provided as the expected output of the training, and a training algorithm based on supervised learning is used to perform training using multiple training input samples and corresponding expected outputs. For example, in order to train a machine learning model trained to perform image segmentation, multiple samples of imaging sensor data can be provided as training input samples, and corresponding pixel masks can be provided as the expected output of the training, and a training algorithm based on supervised learning is used to perform training using multiple training input samples and corresponding expected outputs. In some examples, the same machine learning model can be used for both object detection and image segmentation. In this case, the above two types of expected outputs can be used in parallel during training to train the machine learning model to output both bounding boxes and pixel masks.

[0044] Typically, both object detection and image segmentation can be used. The pixel mask output by the image segmentation machine learning model can be used to determine the scope of one or more anatomical features, and the classification provided by the object detection can be used to determine whether the anatomical feature is of interest. Accordingly, at least one of the two techniques of "object detection" and "image segmentation" can be used to analyze the second imaging sensor data and determine the anatomical features within the second imaging sensor data. For example, the system can be configured to perform image segmentation on the second imaging sensor data and determine the scope of one or more anatomical features based on a pixel mask or multiple pixel masks output by the image segmentation. Alternatively, or in addition, the system can be configured to perform object detection on the second imaging sensor data to identify at least one anatomical feature and determine one or more features of interest among the identified (one or more) anatomical features.

[0045] In some examples, features used to determine one or more anatomical features (e.g., at least one feature of interest) may be limited to a specific set of features. For example, the system may be configured to determine at least one feature of interest among one or more anatomical features of the surgical site. For example, the system may be configured to perform object detection to identify at least one of a blood vessel, a branch point of a blood vessel, and a tumor within the second imaging sensor data as an anatomical feature (of interest). Accordingly, a machine learning model trained to perform object detection may be trained to detect at least one of a blood vessel, a branch point of a blood vessel, and a tumor in the imaging sensor data. Similarly, a machine learning model trained to perform image segmentation may be trained to perform image segmentation on at least one of a blood vessel, a branch point of a blood vessel, and a tumor in the imaging sensor data. Accordingly, one or more anatomical features (of interest) may be determined based on the output of one or more machine learning models, and one or more machine learning models are trained to output information representing a specific feature, such as the aforementioned blood vessel, branch point, or tumor, such as a bounding box or a pixel mask.

[0046] In various examples of the present disclosure, the first imaging sensor data is based on fluorescence imaging, and with respect to spatially varying noise filtering, (only) anatomical features that carry fluorophores (and thus emit fluorescence emissions) are of interest. In other words, one or more anatomical features may be one or more blood vessels, or one or more other types of tissues (such as tissues of a tumor). Accordingly, at least one feature of interest may be at least one blood vessel or other type of tissue that emits fluorescence emissions. To determine whether an anatomical feature such as a blood vessel is of interest, the determined range of the anatomical feature may be cross-referenced with the first imaging sensor data, for example, to determine the blood vessels or tissues that carry fluorophores and are therefore of interest with respect to the proposed noise filtering method. In other words, the system may be configured to determine at least one feature of interest in one or more anatomical features based on the first imaging sensor data. For example, the system may be configured to compare the determined range of one or more anatomical features with the presence of fluorescence emissions in the first imaging sensor data, and determine that the anatomical feature is of interest if the fluorescence emissions (e.g., a sufficient number of fluorescence emissions) intersect the range of the corresponding anatomical feature.

[0047] In some cases, the fluorescence emission is not visible in every frame of the first imaging sensor data. Therefore, multiple frames of the first imaging sensor data can be used when determining whether an anatomical feature is of interest. For example, the system can be configured to determine at least one feature of interest in one or more anatomical features based on multiple frames of the first imaging sensor data covering a predefined time interval or two or more predefined time points. For example, the predefined time interval can be a time interval extending into the past (relative to the time when the determination is made), for example, to at least 5 seconds in the past, or at least 10 seconds, or at least 30 seconds in the past. Similarly, the two or more predefined time points can include a current time point and a past time point, for example, a time point to at least 5 seconds in the past, or at least 10 seconds, or at least 30 seconds in the past.

[0048] Once one or more anatomical features or at least one feature of interest are determined, they can be used as a basis for spatially varying noise filtering. In other words, the system is configured to apply spatially varying noise filtering to the first imaging sensor data based on the range of one or more anatomical features of the surgical site, for example, based on the range of at least one feature of interest. In practice, the first imaging sensor data can be subdivided into different parts, and different noise filters are applied to different parts. For example, a pixel of the first imaging sensor data can be assigned to one of two or more different parts. Noise filtering can be applied to the corresponding pixel based on the part to which the corresponding pixel is assigned.

[0049] In the basic example, two different parts are used - a first part including at least one feature of interest (or one or more anatomical features, if the concept of feature of interest is not used), and a second part containing the rest of the first imaging sensor data. In other words, the system can be configured to subdivide the first imaging sensor data into a first part and a second part. The first part can be based on the range of at least a subset of one or more anatomical features of the surgical site (e.g., based on the range of at least one feature of interest). For example, the first part can include the range of at least a subset of one or more anatomical features of the surgical site (e.g., the range of at least one feature of interest). The second part can include the rest of the first imaging sensor data. For example, pixels that are part of the range of at least a subset of one or more anatomical features of the surgical site, for example, pixels that are part of at least one feature of interest, can be assigned to the first part, and the remaining pixels can be assigned to the second part. In the first part, the pixels can be assigned to different coherent subparts (this will be explained below). The system can be configured to apply a first noise filter to the first part, and a different second noise filter to the second part. In other words, the noise filtering can vary between the first part and the second part, i.e., the noise filtering applied to the first part can be different from the noise filtering applied to the second part.

[0050] The second portion may be considered of less interest to the surgeon because it may be considered the "background", while at least one feature of interest defining the first portion may be considered the "foreground". In general, the second portion, i.e., the background, may be expected to be mostly static and contain mostly soft transitions between different types of tissue (due to the composition of the background and / or due to loss of focus). Therefore, a spatial low-pass filter may be applied to the second portion of the first imaging sensor data. In other words, the second noise filter may be configured to apply a spatial low-pass filter to the second portion of the first imaging sensor data. For example, a two-dimensional Fourier transform may be used to convert individual pixels into a frequency space, where a spatial low-pass filter may be applied.

[0051] In various examples of the present disclosure, it is assumed that the first imaging sensor data is based on fluorescence imaging. Figure 1cAs discussed, fluorescence images tend to have a homogeneous intensity profile within each blood vessel, with little or no variation along a horizontal axis (e.g., an axis orthogonal to the flow of blood) and a gradient along a vertical axis (e.g., an axis along the flow of blood). Similar assumptions can be made about other types of first imaging sensor data, such that a known intensity pattern can be assumed in many cases. Such a known intensity pattern can be used to define a noise filter for the first portion. For example, the first noise filter can be configured to apply a predefined intensity pattern to the first portion of the first imaging sensor data. For example, the predefined intensity pattern can be unique to anatomical features included in the first portion.

[0052] In general, microscopes have a small field of view due to the magnification they provide. In this field of view, in a basic approximation, the intensity of the fluorescent emissions emitted by fluorophores present in the blood vessels can be considered to be uniform across the blood vessels. Accordingly, the predefined intensity pattern can be configured to apply a uniform intensity distribution to the first portion or a coherent subportion of the first portion. For example, a uniform intensity distribution (i.e., the same intensity) can be applied to the entire first portion. Alternatively, at each anatomical feature (of interest), a separate uniform intensity distribution can be applied. Furthermore, a separate uniform intensity distribution can be applied to portions of anatomical features. Alternatively, the predefined intensity pattern can be configured to apply a uniform intensity distribution along the horizontal axis of the coherent subportion of the first portion and to apply an intensity gradient along the vertical axis of the coherent subportion of the first portion. Depending on the granularity used, the first filter can be applied to the entire first portion or to a coherent subportion of the first portion. Accordingly, if a finer granularity is used, the first portion can be subdivided into coherent subportions. In other words, the system can be configured to subdivide the first portion into a plurality of coherent subportions based on the range of at least a subset of one or more anatomical features. About Figure 1d shows this breakdown. Figure 1d represents the second imaging sensor data, but it is used to illustrate the concept because in Figure 1d The relevant sub-parts can be more clearly defined in .

[0053] As described above, coherent sub-portions at different granularity levels can be used. For example, each blood vessel can be considered as a separate coherent sub-portion. For example, blood vessel 14 can be considered as a coherent sub-portion, and blood vessel 16 can be considered as another coherent sub-portion. Since blood vessels are generally interconnected structures with many branches, a finer granularity can be selected for subdivision. For example, the coherent sub-portions of a blood vessel can be defined by the branching points of the blood vessel. For example, blood vessel 14 can be subdivided into coherent sub-portions 14a; 14b and 14c at the branching points between sub-portion 14a and sub-portions 14b and 14c. In other words, the system can be configured to subdivide the first portion into multiple coherent sub-portions based on the branching points of one or more blood vessels. Depending on the selected granularity, the first filter can be applied to each sub-portion separately. For example, the first filter can be applied to each blood vessel separately or to each coherent sub-portion of the blood vessel separately. In fact, the system can be configured to apply the first noise filter to the coherent sub-portions separately.

[0054] Depending on which type of predefined intensity distribution is used, different measurements may be used as references to determine the intensity reference of the corresponding predefined intensity distribution. For example, the system may be configured to determine the average or median intensity of pixels of the first imaging sensor data belonging to the first portion or a coherent subportion of the first portion, and determine the intensity of the uniform intensity distribution based on the average or median intensity of pixels belonging to the first portion or a coherent subportion of the first portion. If a gradient along the longitudinal axis is used, the intensity of the pixels at each end of the vessel or a coherent subportion of the vessel may be determined, and the intensity value of the gradient may be determined based on the intensity of the pixels at each end of the vessel or a coherent subportion of the vessel.

[0055] The result of the denoising can be displayed on a display device of the surgical microscope system, for example, a visual display 130a and / or an auxiliary display 130b. For example, the system can be configured to generate a display signal for a display device 130 of the surgical microscope system based on at least the filtered first imaging sensor data. For example, the system can be configured to generate a digital view of the surgical site based (only) on the filtered first imaging sensor data, for example, a digital view that isolates and displays fluorescence emissions. Alternatively, the system can be configured to generate a digital view of the surgical site based on the filtered first imaging sensor data and based on the (filtered or unfiltered) second imaging sensor data, for example, a composite digital view. For example, the system can be configured to generate a composite digital view of the surgical site based on the filtered first imaging sensor data and based on the second imaging sensor data. For example, the fluorescence emissions can be included in the composite digital view as a pseudo-color representation (e.g., using a single color that is clearly visible relative to the second imaging sensor data). The system can be configured to generate a display signal based on the digital view, for example, based on the composite digital view. For example, the display signal can be a signal for driving (e.g., controlling) the display device 130. For example, the display signal can include video data and / or control instructions to drive the display. For example, the display signal may be provided via one of the one or more interfaces 112 of the system. Accordingly, the system 110 may include a video interface 112 adapted to provide the display signal to a display 130 of the microscope system 100.

[0056] In the proposed microscope system, at least one optical imaging sensor is used to provide imaging sensor data. Accordingly, the first optical imaging sensor 122 is configured to generate first imaging sensor data. Optionally, the second optical imaging sensor 124 or another type of sensor is configured to generate second imaging sensor data. For example, the (one or more) optical imaging sensors 122; 124 of the microscope 120 may include or may be imaging sensors based on APS (active pixel sensor) or CCD (charge coupled device). For example, in an APS-based imaging sensor, light is recorded at each pixel using a photodetector and an active amplifier of the pixel. APS-based imaging sensors are typically based on CMOS (complementary metal oxide semiconductor) or S-CMOS (scientific CMOS) technology. In a CCD-based imaging sensor, incident photons are converted into electron charges at the semiconductor-oxide interface and then moved between the capacitive bins of the imaging sensor through the circuit of the imaging sensor for imaging. The processing system 110 may be configured to obtain (i.e., receive or read out) corresponding imaging sensor data from each sensor 122; 124. The imaging sensor data may be obtained by receiving the imaging sensor data from the respective sensors (e.g., via interface 112), by reading the imaging sensor data from a memory of the respective sensors (e.g., via interface 112), or by reading the imaging sensor data from storage device 116 of system 110, e.g., after the imaging sensor data has been written to storage device 116 by a sensor or another system or processor.

[0057] One or more interfaces 112 of the system 110 may correspond to one or more inputs and / or outputs for receiving and / or transmitting information within a module, between modules, or between modules of different entities, which information may be digital (bit) values ​​according to a specified code. For example, one or more interfaces 112 may include an interface circuit configured to receive and / or transmit information. One or more processors 114 of the system 110 may be implemented using one or more processing units, one or more processing devices, or any means for processing (such as a processor, a computer, or a programmable hardware component that can be operated using corresponding adaptive software). In other words, the described functions of one or more processors 114 may also be implemented in software and then executed on one or more programmable hardware components. These hardware components may include a general-purpose processor, a digital signal processor (DSP), a microcontroller, etc. One or more storage devices 116 of the system 110 may include at least one element of the following group of computer-readable storage media: such as magnetic or optical storage media, such as a hard disk drive, a flash memory, a floppy disk, a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electronically erasable programmable read-only memory (EEPROM), or a network memory.

[0058] With respect to the proposed concept or one or more examples described above or below (e.g. Figures 2 to 4 ), mentions more details and aspects of the system and surgical microscope system. The system and surgical microscope system may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples described above or below.

[0059] Figure 2 A flowchart of an example of a corresponding method for a surgical microscope system is shown. For example, the method may be Figure 1a to Figure 1d The method includes obtaining 210 first imaging sensor data of a view on a surgical site from a first optical imaging sensor of a microscope of the surgical microscope system. The method includes obtaining 220 second imaging sensor data of a view on the surgical site from a second sensor of the microscope. The method includes determining 230 an extent of one or more anatomical features of the surgical site based on the second imaging sensor data. The method includes applying 240 a spatially varying noise filter to the first imaging sensor data based on the extent of the one or more anatomical features of the surgical site.

[0060] For example, in Figure 1a-1d The features introduced in the system and / or surgical microscope system can also be introduced into Figure 2 The corresponding method.

[0061] With respect to the proposed concept or one or more examples described above or below (e.g. Figures 1a to 1d , 3 to 4), mention more details and aspects of the method. The method may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples described above or below.

[0062] Various examples of the present disclosure relate to concepts for spatially selective (ie, spatially varying) noise filtering, such as spatially selective noise filtering of fluorescence images using information gathered from white light images. In short, the concepts presented may be related to selective noise filtering.

[0063] In particular, the proposed concept can be used to filter out noise from a fluorescence image using information from a white light image. By evaluating the white light image, the fluorescence image can be subdivided into a background portion and a fluorescence region portion. In practice, different filters can be applied to different portions of the image. For example, a high pass filter can be applied to the background, and a low pass filter can be applied to the fluorescence region. By reducing the noise on the fluorescence image, the image quality of the fluorescence image can be improved.

[0064] In particular, in some fluorescence imaging modalities, the fluorescence emission is known to be limited to vasculature and accordingly, (aggressive) filters can be applied to reduce the noise in the background. Noise filtering can be made more efficient when additional information about the imaging sensor data is available. For example, if the image can be segmented into regions with homogeneous fluorescence intensity (e.g., coherent sub-fractions), it may be more efficient to filter each segment individually while maintaining the sharpness of the segment boundaries. In more complex cases, when it is known that the intensity within a segment follows a specific pattern, it may be possible to fit / filter the signal. In fact, the proposed concept is based on the use of additional information in the filtering process, where the additional information is extracted from a simultaneously captured white light image or any other image. Furthermore, prior knowledge about the expected spatial pattern of the fluorescence signal can be used for a more efficient filtering process.

[0065] Figure 3 A schematic diagram of applying noise filtering to a noisy fluorescence image 310 (e.g., first imaging sensor data) is shown. In some systems, a general noise filter 320 is applied to the noisy fluorescence image, resulting in a blurred filtered image 330, i.e., the filtered image has low contrast and soft edges. In the proposed concept, a white light image 340 (e.g., second imaging sensor data) is segmented 350 into different parts 360 (e.g., coherent sub-parts) and a segment-by-segment noise filter 370 is applied, resulting in a higher contrast filtered image 380 with sharper edges.

[0066] For example, ICG (indocyanine green) is commonly used for imaging of blood vessels. The fluorescence intensity within each vessel can be considered to be constant within the vessel, and the shape of the vessel can be extracted from the white light image. A simple filtering model can be implemented by calculating the average fluorescence intensity within the vessel and assigning its value to all vessel pixels. A more complex model can assume a smooth variation (e.g., gradient) along the length (vertical axis) of the vessel, while being constant along the width (horizontal axis). In this case, a 2D fitting model can be applied to assign values ​​within the vessel pixels.

[0067] For background tissue, it is not possible to assume a constant value for all pixels, but it can be assumed that high spatial frequencies are not present due to the typical smooth appearance of tissue without sharp boundaries. Accordingly, a spatial low-pass filter can be applied. This spatially selective / variable filtering model can lead to a significant increase in sensitivity, since fluorescence values ​​can be calculated from hundreds or thousands of pixels. Thus, signals that are one or two orders of magnitude weaker than in the unfiltered image can be effectively detected. Thus, 1 / 10 or 1 / 100 of the fluorophore dose can be administered.

[0068] A more general implementation of the proposed concept involves analysis of white light images to compute a 2D spatial spectrum representing the anatomical properties of the tissue and thereby gain knowledge about spatial patterns that are not expected to appear in fluorescence images and that may potentially appear in turn in fluorescence images.

[0069] The spatially selective filtering concept can be relevant to image enhancement, e.g. for microsurgery (vessels and tumors). It can be based on the use of software and / or machine learning.

[0070] With respect to the proposed concept or one or more examples described above or below (e.g. Figures 1a to 2 ), mentions more details and aspects of the concept of spatially selective noise filtering. The concept of spatially selective noise filtering may include one or more additional optional features, corresponding to one or more aspects of the proposed concept or one or more examples described above or below.

[0071] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0072] Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where the blocks or devices correspond to method steps or features of method steps. Similarly, aspects described in the context of method steps also represent a description of the corresponding blocks or items or features of the corresponding apparatus.

[0073] Some embodiments relate to a microscope comprising the Figure 3Alternatively, the microscope may be a system as described in one or more of FIGS. Figure 3 Part of or connected to one or more of the described systems. Figure 4 A schematic diagram of a system 400 configured to perform the methods described herein is shown. System 400 includes a microscope 410 and a computer system 420. Microscope 410 is configured to capture images and is connected to computer system 420. Computer system 420 is configured to perform at least a portion of the methods described herein. Computer system 420 may be configured to execute a machine learning algorithm. Computer system 420 and microscope 410 may be separate entities, but may also be integrated in a common housing. Computer system 420 may be part of a central processing system of microscope 410 and / or computer system 420 may be part of a subcomponent of microscope 410, such as a sensor, actuator, camera, or lighting unit of microscope 410.

[0074] The computer system 420 may be a local computer device (e.g., a personal computer, a laptop, a tablet computer, or a mobile phone) having one or more processors and one or more storage devices, or a distributed computer system (e.g., a cloud computing system having one or more processors and one or more storage devices distributed at different locations, for example, at a local client and / or one or more remote server farms and / or data centers). The computer system 420 may include any circuit or combination of circuits. In one embodiment, the computer system 420 may include one or more processors, which may be of any type. As used herein, a processor may refer to any type of computing circuit, such as, but not limited to, for example, a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA) of a microscope or microscope assembly (e.g., a camera) or any other type of processor or processing circuit. Other types of circuits that may be included in the computer system 420 may be custom circuits, application specific integrated circuits (ASICs), or the like, such as one or more circuits (such as communication circuits) for wireless devices such as mobile phones, tablet computers, laptops, two-way radios, and similar electronic systems. The computer system 420 may include one or more storage devices, which may include one or more storage elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard disk drives, and / or one or more drives for handling removable media, such as compact disks (CDs), flash memory cards, digital video disks (DVDs), etc. The computer system 420 may also include a display device, one or more speakers, a keyboard and / or controller, which may include a mouse, a trackball, a touch screen, a voice recognition device, or any other device that allows a system user to input information to and receive information from the computer system 420.

[0075] Some or all of the method steps may be performed by (or using) a hardware device such as, for example, a processor, a microprocessor, a programmable computer or an electronic circuit. In some embodiments, the most important one or more method steps may be performed by such a device.

[0076] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. The implementation may be performed using a non-transitory storage medium, such as a digital storage medium, for example a floppy disk, DVD, Blu-ray, CD, ROM, PROM and EPROM, EEPROM or flash memory, which has electronically readable control signals stored thereon, which cooperate with (or can cooperate with) a programmable computer system in order to perform the corresponding method. Thus, the digital storage medium may be computer readable.

[0077] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0078] Generally, the embodiments of the present invention can be implemented as a computer program product with a program code, which is operable to perform one of the above methods when the computer program product runs on a computer.For example, the program code can be stored on a machine-readable carrier.

[0079] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0080] In other words, therefore, an embodiment of the inventive method is a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0081] Therefore, a further embodiment of the present invention is a storage medium (or a data carrier, or a computer-readable medium) comprising a computer program stored thereon for performing one of the methods described herein when executed by a processor. The data carrier, the digital storage medium or the recorded medium is typically tangible and / or non-transitory. A further embodiment of the present invention is an apparatus as described herein comprising a processor and a storage medium.

[0082] Therefore, a further embodiment of the invention is a data stream or a sequence of signals representing the computer program for performing one of the methods described in the invention.For example, the data stream or the sequence of signals may be configured to be transmitted via a data communication connection, for example, via the Internet.

[0083] A further embodiment comprises a processing device, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0084] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0085] A further embodiment according to the invention comprises an apparatus or system configured to transmit (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. For example, the receiver may be a computer, a mobile device, a storage device, etc. For example, the apparatus or system may comprise a file server for transmitting the computer program to the receiver.

[0086] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may be coordinated with a microprocessor to perform one of the methods described herein. Generally, the method is preferably performed by any hardware device.

[0087] Embodiments may be based on the use of machine learning models or machine learning algorithms. Machine learning refers to algorithms and statistical models that a computer system can use to perform specific tasks without explicit instructions, rather than relying on models and reasoning. For example, in machine learning, data transformations inferred from analysis of historical and / or training data may be used instead of rule-based data transformations. For example, a machine learning model or machine learning algorithm may be used to analyze the content of an image. In order for a machine learning model to analyze the content of an image, a training image may be used as input and training content information may be used as output to train the machine learning model. By training a machine learning model using a large number of training images and / or training sequences (e.g., words or sentences) and associated training content information (e.g., labels or annotations), the machine learning model "learns" to recognize the content of the image, so that the machine learning model can be used to recognize the content of the image not included in the training data. The same principle can also be used for other types of sensor data: by training a machine learning model using training sensor data and expected outputs, the machine learning model "learns" the conversion between sensor data and output, which can be used to provide output based on non-training sensor data provided to the machine learning model. The provided data (e.g., sensor data, metadata, and / or image data) may be preprocessed to obtain a feature vector, which may be used as input to the machine learning model.

[0088] Machine learning models can be trained using training input data. The above example uses a training method called "supervised learning". In supervised learning, a machine learning model is trained using multiple training samples, where each sample can include multiple input data values ​​and multiple expected output values, i.e., each training sample is associated with an expected output value. By specifying the training samples and the expected output values, the machine learning model "learns" which output value to provide based on input samples that are similar to the samples provided during training. In addition to supervised learning, semi-supervised learning can also be used. In semi-supervised learning, some of the training samples lack corresponding expected output values. Supervised learning can be based on supervised learning algorithms (such as classification algorithms, regression algorithms, or similarity learning algorithms). Classification algorithms can be used when the output is restricted to a limited set of values ​​(categorical variables), i.e., the input is classified into a limited set of values. Regression algorithms can be used when the output can be any value (within a certain range). Similarity learning algorithms can be similar to classification and regression algorithms, but it is based on learning from examples using a similarity function, which measures the degree of similarity or correlation between two objects. In addition to supervised or semi-supervised learning, unsupervised learning can also be used to train machine learning models. In unsupervised learning, one may (only) be provided with input data, and an unsupervised learning algorithm may be used to find structure in the input data (e.g., find commonalities in the data by grouping or clustering the input data). Clustering is the assignment of input data, which includes multiple input values, into subsets (clusters) so that input values ​​in the same cluster are similar according to one or more (predefined) similarity criteria, but dissimilar to input values ​​included in other clusters.

[0089] Reinforcement learning is the third group of machine learning algorithms. In other words, reinforcement learning can be used to train machine learning models. In reinforcement learning, one or more software actors (called "software agents") are trained to take actions in an environment. Based on the actions taken, rewards are calculated. Reinforcement learning is based on training one or more software agents to select actions that increase the cumulative reward, resulting in the software agents becoming better at a given task (as evidenced by increasing rewards).

[0090] In addition, some techniques can be applied to some machine learning algorithms. For example, feature learning can be used. In other words, a machine learning model can be trained at least in part using feature learning, and / or a machine learning algorithm can include a feature learning component. Feature learning algorithms, which can also be called representation learning algorithms, can preserve the information in the input but can also transform it in a useful way, usually as a preprocessing step before making a classification or prediction. For example, feature learning can be based on principal component analysis or cluster analysis.

[0091] In some examples, anomaly detection (i.e., outlier detection) can be used, the goal of which is to identify input values ​​that are suspicious because they are significantly different from the majority of inputs or training data. In other words, a machine learning model can be trained at least in part using anomaly detection, and / or a machine learning algorithm can include an anomaly detection component.

[0092] In some examples, a machine learning algorithm may use a decision tree as a prediction model. In other words, a machine learning model may be based on a decision tree. In a decision tree, observations about an item (e.g., a set of input values) may be represented by branches of the decision tree, and output values ​​corresponding to the item may be represented by leaves of the decision tree. A decision tree may support both discrete and continuous values ​​as output values. If discrete values ​​are used, the decision tree may be represented as a classification tree, and if continuous values ​​are used, the decision tree may be represented as a regression tree.

[0093] Association rules are another technique that can be used in machine learning algorithms. In other words, a machine learning model can be based on one or more association rules. Association rules are created by identifying relationships between variables in a large amount of data. Machine learning algorithms can identify and / or utilize one or more relationship rules that represent knowledge learned from the data. Rules can be used to store, manipulate, or apply knowledge.

[0094] Machine learning algorithms are typically based on machine learning models. In other words, the term "machine learning algorithm" may refer to a set of instructions for creating, training, or using a machine learning model. The term "machine learning model" may indicate a data structure and / or rule set that represents learned knowledge (e.g., based on training performed by a machine learning algorithm). In an embodiment, the use of a machine learning algorithm may mean the use of an underlying machine learning model (or multiple underlying machine learning models). The use of a machine learning model may mean that the machine learning model and / or the data structure / rule set that is the machine learning model is trained by a machine learning algorithm.

[0095] For example, a machine learning model may be an artificial neural network (ANN). ANN is a system inspired by biological neural networks, such as those found in the retina or brain. ANN comprises a plurality of interconnected nodes and a plurality of connections between nodes, so-called edges. There are usually three types of nodes, input nodes that receive input values, hidden nodes that are (only) connected to other nodes, and output nodes that provide output values. Each node may represent an artificial neuron. Each edge may transfer information from one node to another. The output of a node may be defined as a (non-linear) function of its input (e.g., the sum of its inputs). The input of a node may be used in a function based on the "weight" of the edge or node that provides the input. The weights of nodes and / or edges may be adjusted during the learning process. In other words, the training of an artificial neural network may include adjusting the weights of the nodes and / or edges of the artificial neural network, i.e., achieving the desired output for a given input.

[0096] Alternatively, the machine learning model can be a support vector machine, a random forest model or a gradient boosting model. A support vector machine (i.e., a support vector network) is a supervised learning model with an associated learning algorithm that can be used to analyze data (e.g., classification or regression analysis). A support vector machine can be trained by providing an input with multiple training input values ​​belonging to one of two categories. A support vector machine can be trained to assign new input values ​​to one of the two categories. Alternatively, the machine learning model can be a Bayesian network, which is a probabilistic directed acyclic graph model. A Bayesian network can represent a set of random variables and their conditional dependencies with a directed acyclic graph. Alternatively, the machine learning model can be based on a genetic algorithm, which is a search algorithm and heuristic technique that mimics the natural selection process.

[0097] Reference numerals list

[0098] 10 samples, surgical site

[0099] 12 Background Organization

[0100] 14,16 Blood vessels

[0101] 14a, 14b, 14c Coherent subsections of blood vessels

[0102] 100 Surgical Microscope System

[0103] 105 base unit

[0104] 110 system

[0105] 112 interfaces

[0106] 114 processors

[0107] 116 Storage Devices

[0108] 120 Microscope

[0109] 122 Optical Imaging Sensor

[0110] 124 sensor, optical imaging sensor

[0111] 130a Visual Display

[0112] 130b Auxiliary display

[0113] 140 Arm

[0114] 210 Obtaining first imaging sensor data

[0115] 220 Obtaining second imaging sensor data

[0116] 230 Determine the extent of one or more anatomical features

[0117] 240 Apply spatially varying noise filtering

[0118] 310 Noisy fluorescence image

[0119] 320 Universal Noise Filter

[0120] 330 Blurred filtered image

[0121] 340 White light images

[0122] 350 split

[0123] 360 degree segmentation

[0124] 370 Segment-by-segment noise filter

[0125] 380 filtered image

[0126] 400 system

[0127] 410 Microscope

[0128] 420 Computer Systems

Claims

1. A system (110; 420) for a surgical microscope system (100; 400), the system comprising one or more processors (114) and one or more storage devices (116), wherein the system is configured to: obtaining first imaging sensor data of a view on a surgical site (10) from a first optical imaging sensor (122) of a microscope (120; 410) of a surgical microscope system; obtaining second imaging sensor data of a view on the surgical site (10) from a second sensor (124) of the microscope (120); determining an extent of one or more anatomical features of the surgical site based on the second imaging sensor data; A spatially varying noise filter is applied to the first imaging sensor data based on the extent of one or more anatomical features of the surgical site. 2 . The system of claim 1 , wherein the first imaging sensor data is based on fluorescence imaging.

3. A system according to any one of claims 1 or 2, wherein the second imaging sensor data is based on reflectance imaging.

4. A system according to any one of claims 1 to 3, wherein the system is configured to determine at least one feature of interest among one or more anatomical features of the surgical site and apply spatially varying noise filtering based on the range of the at least one feature of interest.

5. The system of claim 4, wherein the one or more anatomical features are one or more blood vessels, and wherein at least one feature of interest is at least one blood vessel that emits fluorescent emissions.

6. A system according to any one of claims 4 or 5, wherein the system is configured to determine at least one feature of interest among the one or more anatomical features based on the first imaging sensor data.

7. A system according to any one of claims 4 or 5, wherein the system is configured to determine at least one feature of interest among one or more anatomical features based on multiple frames of the first imaging sensor data covering a predefined time interval or two or more predefined time points.

8. A system according to any one of claims 1 to 7, wherein the system is configured to subdivide the first imaging sensor data into a first portion and a second portion, the first portion being based on the extent of at least a subset of one or more anatomical features of the surgical site, and the system is configured to apply a first noise filter to the first portion and a different second noise filter to the second portion.

9. The system of claim 8, wherein the system is configured to subdivide the first imaging sensor data such that the first portion includes a range of at least a subset of one or more anatomical features of the surgical site and the second portion includes the remainder of the first imaging sensor data.

10. The system of any one of claims 8 or 9, wherein the first noise filter is configured to apply a predefined intensity pattern to the first portion of the first imaging sensor data.

11. The system of claim 10, wherein the predefined intensity pattern is configured to apply a uniform intensity distribution to the first portion or a coherent sub-portion of the first portion, Or wherein the predefined intensity pattern is configured to apply a uniform intensity distribution along a transverse axis of the coherent sub-portion of the first portion and to apply an intensity gradient along a longitudinal axis of the coherent sub-portion of the first portion.

12. A system according to any one of claims 8 to 11, wherein the system is configured to subdivide the first portion into a plurality of coherent sub-portions based on the extent of at least a subset of the one or more anatomical features, and to apply a first noise filter to each of the coherent sub-portions.

13. The system of claim 12, wherein the one or more anatomical features are one or more blood vessels, wherein the system is configured to subdivide the first portion into a plurality of coherent sub-portions based on branch points of the one or more blood vessels.

14. The system of any one of claims 8 to 13, wherein the second noise filter is configured to apply a spatial low pass filter to the second portion of the first imaging sensor data.

15. A system according to any one of claims 1 to 14, wherein the system is configured to determine the extent of one or more anatomical features of the surgical site using a machine learning model, the machine learning model being trained to perform image segmentation and / or object detection.

16. The system of any one of claims 1 to 15, wherein the system is configured to determine the extent of the one or more anatomical features of the surgical site based on at least one of a characteristic color spectrum and a characteristic shape of the one or more anatomical features of the surgical site.

17. A system according to any one of claims 1 to 16, wherein the system is configured to generate a display signal for a display device (130) of the surgical microscope system based on at least the filtered first imaging sensor data.

18. The system of claim 17, wherein: The system is configured to generate a composite digital view of the surgical site based on the filtered first imaging sensor data and based on the second imaging sensor data, and to generate a display signal based on the composite digital view.

19. A method for a surgical microscope system, the method comprising: obtaining (210) first imaging sensor data of a view on a surgical site from a first optical imaging sensor of a microscope of a surgical microscope system; obtaining (220) second imaging sensor data of a view on the surgical site from a second sensor of the microscope; determining ( 230 ) an extent of one or more anatomical features of the surgical site based on the second imaging sensor data; A spatially varying noise filter is applied (240) to the first imaging sensor data based on the extent of one or more anatomical features of the surgical site.

20. A computer program having a program code for executing the method according to claim 19 when the computer program is executed on a processor.

Images