Imaging scanner and optical scanner in a medical device

Abstract

A medical system comprises a measurement device. The measurement device comprises an imaging scanner, such as an ultrasound scanner, configured to generate an image of a first region of interest, and an optical scanner comprising at least one light emitter configured to emit light into a second region of interest and at least one light detector configured to detect the light that has interacted with a tissue in the second region of interest, to generate optical measurement data. The at least one light emitter and the at least one light detector and the imaging scanner are aligned with each other 10 at least during the measurements, so that the first region of interest overlaps the second region of interest, and the first region of interest is registered with the second region of interest.

Description

[0001] Imaging scanner and optical scanner in a medical device

[0002] FIELD OF THE INVENTION

[0003] The invention relates to a medical system comprising a measurement device. The invention further relates to a medical system configured to calculate a distance. The invention further relates to a combined measurement method. The invention further relates to a computer-implemented measurement method. The invention further relates to a computer-implemented image analysis method. The invention further relates to a method of training a model to classify optical measurement data. The invention further relates to a contact sensor and a method of contact sensing.

[0004] BACKGROUND OF THE INVENTION

[0005] The current primary treatment for cancer is surgical resection, which is often combined with radiation therapy or chemotherapy. In cancer surgery, there is a delicate balance between the complete excision of tumor tissue and the preservation of surrounding vital structures. However, intraoperative differentiation between tumor and healthy tissue can be challenging. Inadequate tumor recognition might result in a positive resection margin, which has been identified as a predictive factor for increased recurrence and decreased survival. Intraoperative tissue type characterization has the potential to reduce positive resection margins and minimize damage to healthy tissue. While frozen section analysis is the current golden standard for intraoperative assessment of the resection margin, it suffers from delayed feedback to the surgeon. A real-time intraoperative technique could assist the surgeon in selecting the optimal resection plane, enabling effective tumor removal while preserving vital healthy tissue. Unfortunately, such a technique is not yet available.

[0006] Diffuse reflectance spectroscopy (DRS) demonstrated promising results in distinguishing between tumor and healthy tissue for several cancer types, including lung, breast, liver, and head and neck. DRS is an optical sensing technique that sends broadband white light into tissue and analyzes the spectral changes after reflection, which is highly specific for different tissue types. DRS is known to discriminate certain tumor types, such as colorectal tumor and breast tumor, from healthy tissue both ex vivo and in vivo. Previous studies focused on identifying tissue at a single depth near the surface, assuming that single-layer homogeneous tissue is present, which is not representative of surgical practice.

[0007] S. Brouwer et al., “Toward complete oral cavity cancer resection using a handheld diffuse reflectance spectroscopy probe,” J. Biomed. Opt. 23, 1-8 (2018) reported on investigation of multiple emitter-to-detector distances, but only to compare the performance of the different distances. Moreover, histopathology is frequently used as the single ground truth for tissue characterization and optical technique validation. However, the histopathology process (fixation and sectioning) can cause deformation of the tissue, resulting in a mismatch between distances measured during DRS data acquisition and the resulting histopathology slice, posing challenges in accurate labeling of measurements. Furthermore, in previous approaches, the labeling of measurements is often performed by simply establishing a tumor threshold in the histopathology slice, based on the rule of thumb that the sampling depth is approximately similar to the distance between the emitting and receiving DRS fiber. However, this method neglects the complex characteristics of the sampling volume of DRS.

[0008] SUMMARY OF THE INVENTION

[0009] In order to provide an improved tool for characterization of a tissue, a medical system is provided comprising a measurement device, the measurement device comprising: an imaging scanner configured to generate an image of a first region of interest; an optical scanner comprising at least one light source configured to emit light into a second region of interest and at least one light detector configured to detect the light that has interacted with a tissue in the second region of interest, to generate optical measurement data, wherein the at least one light emitter and the at least one light detector and the imaging scanner are aligned with each other during the measurements, so that the first region of interest overlaps the second region of interest, and the first region of interest is registered with the second region of interest.

[0010] This way, the image can provide the context of the structural information such as tissue layers, to the optical measurement. By combining the image with the optical measurement data, the optical measurements may become more valuable. For example, the information generated based on the optical measurements and the image together may be more accurate than information based on the optical measurements alone. Moreover, the overall image may help to better position the measurement device to place the optical scanner at a particular location of interest spotted in the image. The operator may detect a point of interest in the ultrasound image, and move the measurement device such that the second region of interest overlaps the point of interest. Then, the optical scanner can perform a measurement at this point of interest to generate highly relevant clinical information. In an example implementation, the first region of interest is planar (a slab) and the second region of interest is a point measurement (or a line measurement), covering a much smaller region than the first region of interest. More generally, the first region of interest may be larger than the second region of interest. In certain embodiments, the second region of interest may be entirely included in the second region of interest.

[0011] The at least one light emitter may be the distal tip of an optical fiber that has a light source at its proximal tip. The light source may be, for example, a light emitting diode (LED) or a laser. The at least one light detector may be the distal tip of an optical fiber that has a light sensor, for example a photosensitive component (e.g. a photodiode or a spectrometer) at its proximal tip. For example, the light source and the light sensor may be light transducers.

[0012] The imaging scanner may be an ultrasound scanner, e.g. an ultrasound probe. The optical scanner may be a diffuse reflectance spectroscopy (DRS) scanner, for example. The combination of these two imaging modalities were found to be particularly complementary to each other, strengthening the clinical relevance of the information generated by the measurement probe.

[0013] In certain embodiments the measurement device is a hand-held device. This allows easy positioning and handling of the measurement device.

[0014] The imaging scanner and the optical scanner may be configured to operate simultaneously. This way, it is ensured that the ultrasound image and the optical measurement data relate to the same tissue, and that the measurements are performed at reproducible relative positions, regardless of movements of the measurement device and motion of a subject that is being imaged. Alternatively, these measurements may be performed shortly one after the other. Alternatively, the imaging scanner may operate continuously, while the device is positioned, and the optical scanner may be triggered when it is correctly aligned to the sample. The system may further comprise a display configured to output the image and an indication of the second region of interest in the image. This may help to verify the position of the optical measurement, so that the optical measurement is performed at a clinically relevant position.

[0015] The system may further comprise a control unit configured to control to perform a diffuse reflectance spectroscopy analysis on the light measurement data. Such diffuse reflectance spectroscopy (DRS) analysis was found to be particularly informative about tissue type characterization, for example tumor detection, and the combination of DRS with ultrasound measurements was found to be particularly fruitful.

[0016] The imaging scanner may comprise an ultrasound scanner, wherein the image generated by the imaging scanner comprises an ultrasound image.

[0017] The optical scanner may comprise a plurality of light source to light detector distances. This way, measurements relating to different tissue depths can be performed, because on average the greater the distance from the light source to the light detector, the further into the tissue the light penetrates before being reflected into the light detector.

[0018] The system may be configured to sequentially or simultaneously activate different pairs of one light emitter of the at least one light emitter and one light detector of the at least one light detector, wherein the different pairs are associated with different light emitter to light detector distances. This way different sampling depths may be realized.

[0019] The optical scanner may comprise at least a light source to light detector distance of at most 2 millimeters; and a light source to light detector distance of at least 6 millimeters. This way, measurements for relatively large depth variation may be performed. The optical scanner may comprise at least one light source to light detector distance that is larger than 2 millimeters and smaller than 6 millimeters. This way, a finetuned depth resolution may be obtained. The optical scanner may comprise at least 6 source to detector distances. This way, an even more fine-tuned depth resolution may be obtained.

[0020] The optical scanner may be aligned in a center of the imaging scanner, so that the second region of interest is registered to a middle of the first region of interest. This provides a good overview of the situation on both sides of the second region of interest. In this case the middle of the first region of interest refers to in the middle measured in one dimension only (parallel to the detector surface). In the other dimension the second region of interest may be near or at an edge of the first region of interest.

[0021] The optical scanner may comprise a plurality of light emitters interleaved with a plurality of light detectors. The light emitters and light detectors may be aligned in a row, for example. This way, the second region of interest may comprise measurement areas of multiple point measurements. This way, the second region of interest may cover a larger portion of the first region of interest. The length of the second region of interest, measured along the scanner device surface, could be as large or larger than the first region of interest, by providing interleaved light emitters and light detectors along the length of the image detector, e.g. along the ultrasound scanner.

[0022] The system may further comprise a control unit. The control unit may be configured to control operation of the components of the system. For example the control unit may comprise a processor system and a memory. This may help to perform control operations to control the components of the medical system, and / or to perform processing operations on data generated by e.g. the imaging scanner and the optical scanner.

[0023] The control unit may be configured to estimate a measurement depth of the optical scanner based on the image. The image may help to learn more about the depth of the optical signal’s penetration into the tissue, and the size of the second region of interest. The structural information found in the (ultrasound) image may be used to estimate the measurement depth of the optical scanner. This way, the measurements of the optical scanner become more useful, because it is important to know at what depth a particular tissue type is detected.

[0024] The control unit may be configured to detect a transition from a first tissue layer with a first tissue type to a second tissue layer with a second tissue type, based on the image. The transition may be particularly well visible in the image, for example an ultrasound image. The control unit may alternatively be configured to detect the transition based on the image and the optical measurement data. The combination of the ultrasound image and the optical measurement data were found to provide even more accurate tissue type classification and / or depth estimation of a tissue type.

[0025] The system may comprise a learned model configured to combine the image and the optical measurement data, which are registered to each other, to estimate an estimated tissue type classification and / or a depth estimation of a tissue type. This provides an efficient way to extract relevant and accurate information from the image and optical measurement data.

[0026] The learned model may be configured to estimate, for a particular measurement comprising an image generated by the imaging scanner and an optical measurement by the optical scanner, a measurement depth associated with at least one light source to light detector distance of the optical scanner, based on the image. This improved estimate of the depth that is associated with each optical measurement may be realized by virtue of the combination with the image.

[0027] The optical scanner may be detachably attachable to the ultrasound scanner. This helps to upgrade legacy ultrasound equipment with the optical scanner set forth. Also, it may facilitate replacement of either the ultrasound scanner or the optical scanner. Moreover, it allows to save cost by selectively using the optical scanner only in cases where the combination of ultrasound with optical measurements has clinical relevance.

[0028] The medical system may further comprise a clip-on unit configured to clip the optical scanner, detachably and at a reproduceable relative position, to the imaging scanner. This facilitates the registration of the first region to the second region, because the imaging scanner and the optical scanner are routinely secured in the same relative position. Still the two modalities can be provided as separately usable devices when not clipped to each other. Moreover, in certain embodiments, even when clipped to each other, the optical scanner can be used without activating the imaging scanner. This also holds the other way round: in certain embodiments, the imaging scanner may also be used without making use of the optical scanner, even when they are clipped to each other.

[0029] The clip-on unit may comprise a sleeve with a slot into which a head of the ultrasound scanner is configured to fit, the clip-on unit further comprising a slot for the optical scanner. The sleeve may be a rigid or semi-rigid sleeve. The imaging scanner fits in its slot at a reproducible position relative to the optical scanner that is located in its own slot. The optical scanner may or may not be removable from its slot. The imaging scanner may be removable from its slot so that the imaging scanner can be used independently from the optical scanner as well. In an alternative implementation, both the optical scanner and the ultrasound scanner are permanently fixed in their slots (for example by means of a glue), so that in effect a single inextricable unit may be created. According to another aspect of the invention, a medical system is configured to calculate a distance from an image scanner to a tissue layer, wherein the system comprises: a learned model configured to process image data generated by an imaging scanner, the learned model comprising a backbone connected to a head architecture; and a control unit configured to estimate a distance from the imaging scanner to a tissue layer by applying the learned model to at least one image generated by the imaging scanner, wherein the head architecture is configured to output a direct estimation of the distance to the tissue layer, based on a feature map generated by the backbone. This head architecture improves the performance of the learned model, by efficiently processing the feature information outputted by the backbone.

[0030] The head architecture may comprise: a horizontal mean pooling layer configured to pool the feature map, and a one-dimensional convolution operating on the horizontal mean pooling layer. These components help to focus the learning on the desired output, which is the distance to the tumor (tumor margin).

[0031] According to another aspect of the invention, a combined measurement method is provided, comprising: providing an imaging scanner aligned with an optical scanner, the optical scanner comprising at least one light source and at least one light detector; generating an image of a first region of interest using the imaging scanner; emitting light into a second region of interest using the light source and detecting the light that has interacted with a tissue in the second region of interest using the light detector, wherein the imaging scanner is aligned with the optical scanner so that the first region of interest overlaps the second region of interest, and the first region of interest is registered with the second region of interest.

[0032] According to another aspect of the invention, a computer-implemented measuring method is provided, comprising: receiving an image of a first region of interest from an imaging scanner; receiving light measurement data corresponding to an optical measurement of a second region of interest, wherein the first region of interest overlaps the second region of interest; accessing parameters of a registration of the first region of interest with the second region of interest; detecting a tissue type in the second region of interest based on the light measurement data and estimating a distance to the tissue type based on the image, and relating the detected tissue type to the estimated distance of the tissue type based on the parameters of the registration.

[0033] According to another aspect of the invention, a computer-implemented image analysis method is provided. The analysis method comprises: receiving an image of a first region of interest, wherein the image is generated by an imaging scanner; and estimating a tumor margin using a learned model configured to process the image generated by the imaging scanner, the learned model comprising a backbone connected to a head architecture, wherein the head architecture outputs a direct estimation of the distance to the tissue layer, based on a feature map generated by the backbone.

[0034] The head architecture may comprise a horizontal mean pooling layer configured to pool a fused feature map of the model and a one-dimensional convolution operating on the horizontal mean pooling layer. The head architecture was found to improve the distance estimation. It will be understood that any optical measurements or optical scanner are not needed for this aspect of the invention.

[0035] According to another aspect of the invention, a method of training a model to classify optical measurement data is provided, the method comprising: accessing a training set comprising pairs of measurements, each pair comprising an image of a first region of interest of a sample and an optical measurement of a second region of interest of the sample, wherein the first region of interest is registered to the second region of interest and the first region of interest overlaps the second region of interest; determining at least one parameter representing an uncertainty regarding measurements stored in the training set; selecting a plurality of combinations of values of the at least one parameter; assigning a classification label to the optical measurement of each pair for each selected combination of values of the at least one parameter, wherein the classification label is extracted from the image of each pair under the assumption of the respective selected combination of values of the at least one parameter; and training a model for each selected combination of values of the at least one parameter, using the classification labels corresponding to the respective selected combination of values of the at least one parameter at target output values for the model.

[0036] This training method, involving exploration of labelling training datasets according to multiple values of parameters that describe uncertain aspects of the measurement apparatus and / or measurement conditions, may help to find out realistic values of the parameters, and may result in improved performance of a learned model.

[0037] The step of assigning the classification label may comprise estimating a distance of a tissue type in the image and estimating a sampling depth of the optical measurement, and comparing the depth of the tissue type in the image to the sampling depth. The training set may be classified based on the comparison: only if the sampling depth exceeds the distance of the distance of the tissue type, then the ground truth classification of the optical measurement may be set to that tissue type.

[0038] The at least one parameter may comprise at least one of: a depth bias that augments an estimated measurement depth of the first measurement modality, a depth uncertainty margin used to exclude measurements where a difference between the estimated distance to the tissue type in the image and the estimated sampling depth is within the depth uncertainty margin; and a width of a subregion of the first region of interest that is used to calculate an average distance to the tissue type in the image; and a scaling factor that scales the width and the depth uncertainty margin.

[0039] These parameters were found to be important to characterize the measurement device and the sampling tissues being investigated, and led to improved performing learned models.

[0040] According to another aspect of the invention, a contact sensor is provided. The contact sensor comprises a light source; a light detector; an optical fiber, wherein a proximal end of the optical fiber is optically coupled to the light source and to the optical detector and the distal tip of the optical fiber comprises a contact surface; and a control unit configured to detect whether the contact surface is in contact with a tissue, based on a signal intensity of a light signal detected at the optical detector while the light source emits light into the optical fiber. This contact sensor may provide particularly reliable result, while being cost effective. Moreover, when integrated with an optical measurement device, the contact sensor may improve the quality of optical measurements by ensuring proper contact with the measured sample.

[0041] The contact sensor may comprise a beam splitter at a proximal tip of the optical fiber, wherein the beam splitter is configured to pass the light from the light source into the optical fiber and to diverge light traveling in the direction from the distal tip of the optical fiber to the proximal tip of the optical fiber to the light detector. This way, for example, the same optical fiber can be used for emitting and detecting of light.

[0042] The contact sensor may comprise a second light detector; and a second optic fiber, wherein a proximal end of the second optical fiber is optically connected to the second light detector, and the distal tip of the second optical fiber comprises a second contact surface. The contact surface of the optical fiber and the second contact surface of the second optical fiber may be aligned in a plane. This provides an improved contact sensing spanning multiple contact points aligned in a plane.

[0043] At least a distal portion of the optical fiber may be included in a housing having an outside surface, and the contact surface may be aligned with the outside surface of the housing, and optionally the second contact surface is also aligned with the outside surface of the housing. This provides a particularly suited construction of the optical contact sensor.

[0044] According to another aspect of the invention, a method of contact sensing is provided. The method comprises: emitting light through an optical fiber through a contact surface at a tip of the optical fiber; detecting an intensity of light received through the contact surface of the tip of the optical fiber; and determining whether the contact surface is in contact with a tissue, based on the detected intensity of the received light. This detected intensity of light proved to be a reliable indicator of proper contact. For example, lower light intensity may be associated with good contact, whereas higher light intensity may be associated with bad or no contact.

[0045] For example, the tissue may have a refractive index that is closer to a refractive index of the optical fiber than to a refractive index of environmental air. Contact with this kind of tissue may be particularly well detected by the method.

[0046] Another aspect of the invention provides a computer program product comprising instructions that cause a control unit to perform any of the methods set forth herein.

[0047] The person skilled in the art will understand that the features described above may be combined in any way deemed useful. Moreover, modifications and variations described in respect of the system may likewise be applied to the method and to the computer program product, and modifications and variations described in respect of the method may likewise be applied to the system and to the computer program product.

[0048] BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In the following, aspects of the invention will be elucidated by means of examples, with reference to the drawings. The drawings are diagrammatic and may not be drawn to scale. Throughout the drawings, similar items may be marked with the same reference numerals.

[0050] Fig. 1A shows an optical probe.

[0051] Fig. 1 B shows an example layout of source-to-detector distances of an optical probe.

[0052] Fig. 10 illustrates sampling depths of light measurements for different source- to-detector distances.

[0053] Fig. 1 D shows a second example layout of multiple sources and multiple detectors of an optical probe.

[0054] Fig. 1 E illustrates exemplary sampling regions of light measurements for different sources and detectors of the second example.

[0055] Fig. 2A shows an example of the probing side of an ultrasound probe with an optical measurement device.

[0056] Fig. 2B shows an optical measurement device clip-on.

[0057] Fig. 2C shows an ultrasound probe with an optical measurement device clip- on, on an excised tissue sample.

[0058] Fig. 2D shows certain spectra for different source-to-detector distances.

[0059] Fig. 3A shows a process of ground truth labeling.

[0060] Fig. 3B illustrates parameters used for ground truth labeling and / or training purposes.

[0061] Fig. 4 shows a flow chart illustrating aspects of a process of training a learned model.

[0062] Fig. 5 illustrates performance of trained models.

[0063] Fig. 6 shows diagrams illustrating relevance of certain parameters.

[0064] Fig. 7 illustrates further diagrams illustrating relevance of certain parameters.

[0065] Fig. 8 illustrates performance of certain trained models. Fig. 9 shows a block diagram illustrating aspects of a medical system comprising a measurement device.

[0066] Fig. 10 shows an ultrasound scanner and an optical measurement device integrated into one solid hand-held measurement device.

[0067] Fig. 11 illustrates a medical system comprising a hand-held measurement device.

[0068] Fig. 12 shows an enlargement of part of the hand-held measurement device shown in Fig. 11.

[0069] Fig. 13 shows a frontal view of a detecting side of a clip-on with an optical scanner.

[0070] Fig. 14 shows a block diagram illustrating aspects of a combined optical / ultrasound system.

[0071] Fig. 15A shows a perspective view another embodiment of a combined ultrasound and optical probe.

[0072] Fig. 15B shows a frontal view of a detecting side of a probe, for example as shown in Fig. 15A.

[0073] Fig. 15C shows an example layout of multiple sources and multiple detectors of an optical probe, for example as shown in Fig. 15A.

[0074] Fig. 15D illustrates exemplary sampling regions of light measurements for different sources and detectors of an optical probe, for example a proas shown in Fig. 15A.

[0075] Fig. 15E shows a functional diagram of an optical measurement system of the combined probe of Fig. 15A.

[0076] Fig. 16A shows an ultrasound image covering a region of interest of a sample comprising a breast tumor.

[0077] Fig. 16B shows an ultrasound image covering a region of interest of a sample comprising a colorectal tumor.

[0078] Fig. 17 shows a diagram illustrating aspects of a learned model with a head architecture.

[0079] Fig. 18 illustrates performance of certain trained models with a head architecture.

[0080] Fig. 19 illustrates predicted tumor margins versus true tumor margins, using a trained model with head architecture. Fig. 20 shows visualization of true and predicted tumor margins for multiple breast (top) and colorectal (bottom) ultrasound images.

[0081] Fig. 21 shows a comparison between tumor margin prediction performance using an indirect segmentation-based approach (according to a related study) and a direct prediction based approach (according to an embodiment of the present invention).

[0082] Fig. 22 shows a flowchart illustrating aspects of a method of a combined measurement method.

[0083] Fig. 23 shows a flowchart illustrating aspects of a computer-implemented measurement method.

[0084] Fig. 24 shows a flowchart illustrating aspects of a computer-implemented image analysis method.

[0085] Fig. 25 illustrates a first example of a contact sensor in a first situation.

[0086] Fig. 26 illustrates the first example of a contact sensor in a second situation.

[0087] Fig. 27 illustrates a second example of a contact sensor.

[0088] Fig. 28 illustrates a third example of a contact sensor.

[0089] Fig. 29 shows a flowchart illustrating aspects of a method of contact sensing.

[0090] Fig. 30 shows a diagram illustrating aspects of a method of combined processing of optical measurement data and ultrasound images.

[0091] Fig. 31 shows true tumor margin versus predicted tumor margin for combined processing of optical measurement data and ultrasound images.

[0092] Fig. 32 shows a side view of an optical fiber.

[0093] Fig. 33 shows a side view of an image scanner and an optical scanner.

[0094] Fig. 34 shows a side view of an oblique image scanner and an optical scanner.

[0095] Fig. 35 shows a side view of an image scanner and an oblique optical scanner.

[0096] Fig. 36 shows a side view of an oblique image scanner and an oblique optical scanner.

[0097] Fig. 37 shows a side view of an image scanner and a parallel optical scanner.

[0098] Fig. 38 shows a perspective view of an image scanner and an optical scanner.

[0099] Fig. 39 shows a perspective view of a retracted image scanner and an optical scanner.

[0100] Fig. 40 shows a perspective view of a retracted image scanner and an oblique optical scanner. DETAILED DESCRIPTION OF EMBODIMENTS

[0101] Certain exemplary embodiments will be described in greater detail, with reference to the accompanying drawings.

[0102] The matters disclosed in the description, such as detailed construction and elements, are provided to assist in a comprehensive understanding of the exemplary embodiments. Accordingly, it is apparent that the exemplary embodiments can be carried out without those specifically defined matters. Also, well-known operations or structures are not described in detail, since they would obscure the description with unnecessary detail.

[0103] In certain embodiments, diffuse reflectance spectroscopy for tumor detection may be performed to examine tissue layers at various depths from the surface of a sample (sample can be e.g. a specimen, patient) using a fiber-array. For example, this may be performed during colorectal and breast cancer surgery or other tumor diagnosis sessions or surgery sessions. For example, this procedure may be performed on excised specimen (ex vivo). This way, the excised tissue may be analyzed. Alternatively, diffuse reflectance spectroscopy devices may be used performed in-vivo, for example on the patient skin or on a surgical plane.

[0104] In the present disclosure, the expression “scanner” may mean a combination of passive components arranged to collect incoming signals (such as optical signals or acoustic signals or electromagnetic signals), and forward the incoming signals to a transducer that converts the incoming signals to an electronic signal. Similarly the scanner may receive signals (e.g. optical signals or acoustic signals or electromagnetic signals) from a transducer and emit those signals.

[0105] In certain embodiments, a fiber-array DRS clip-on is provided. The DRS clip-on may comprise about six different emitter-to-detector distances. Other numbers of emitter-to-detector distances are possible in alternative embodiments. For example, a configuration of at least 3 emitter-to-detector distances may be advantageous.

[0106] The DRS clip-on, or a DRS spectroscopy scanner or optical scanner in general, may comprise a plurality of pairs of light emitting fibers and light receiving fibers. For example, the emitter-to-detector distance of each pair could be identical. The tips of the fibers may be arranged in a line or plane perpendicular to the tips of the fibers. The multiple pairs allow a greater area to be measured without moving the optical scanner. Also, by receiving the light signal from one light emitting fiber at multiple of the light receiving fibers, different sampling depths may be realized.

[0107] For example, the DRS clip-on may comprise about eight fibers (or more) with same distances where four fibers are emitters and four fibers are detectors and they are positioned alternatingly. Other numbers of emitters and detectors are possible in alternative embodiments. For example, a configuration of at least 4 fibers may be advantageous. These configurations provide measurements in a bigger region making the entire surface scanning faster.

[0108] The clip-on may have the capability of attaching it to an ultrasound (US) probe, facilitating simultaneous DRS and US data acquisition with reproducible relative positioning of the ultrasound probe and the DRS fiber-array. Alternatively, the DRS clip- on may be attached to another imaging modality than an ultrasound probe, such as an optical coherence tomography scanner. Throughout this disclosure, the emphasis of the detailed description may be on ultrasound and DRS. However, this is not a limitation. Throughout the present disclosure, ultrasound may be replaced by any suitable alternative medical imaging modality, and DRS may be replaced by any suitable alternative optical measurement technology.

[0109] For example, the US data may be used for accurate ground truth tumor margin measurements. For example, these US data may be labeled by human evaluators to create the ground truth. Having the US and the DRS in one device may help to obtain the best possible co-registered US and DRS data. This may improve the learning of a learned model to interpret the DRS measurements. Also, it may improve the estimation of penetration depth of the optical measurements, as will be elaborated in the present disclosure.

[0110] In certain embodiments, the optical scanner collects data from a plurality of different sampling depths (i.e. , regions at certain depths in the tissue from the surface of the tissue). For example, the scanner collects data from a plurality of different sampling depths, using a fiber-array probe with multiple emitter-to-detector distances (SDD).

[0111] To obtain ground truth distances, histopathology and / or ultrasound may be used, either alone or in combination. In addition an ultrasound (US) scanner integrated with an optical scanner in a single device, may be used to obtain more accurate ground truth distances from the scanner device to the tumor (or to any particular tissue type that can be detected by US) at the time of the optical measurements. For this reason (but also for other reasons, mentioned elsewhere herein), certain embodiments comprise a multimodality tool that generates DRS data (or other kind of optical data) for tissue type characterization and US imaging for extracting spatial information. Also, the tissue type characterization and / or generation of spatial information can be improved by combining the DRS data and the US data, for example by means of a learned model. In certain embodiments, the complex characteristics of the sampling volume may be taken into account by exploring influencing factors and by conducting a grid search optimization for the labeling process.

[0112] Certain advantageous embodiments comprise a fiber-array DRS probe with a plurality of emitter-to-detector distances for retrieving information at multiple depths from the tissue surface. For example, six emitter-to-detector distances may be implemented.

[0113] Certain advantageous embodiments comprise a hybrid DRS-US probe that enables simultaneous acquisition of co-registered DRS spectra and US images using a single probe, by incorporating optical fibers of the DRS probe in a clip-on for the ultrasound transducer. The clip-on is configured so that the DRS probe can be fixedly clipped onto the ultrasound transducer. Preferably the clip-on is detachable so that the DRS probe can be removed from the ultrasound transducer when it is no longer needed. The hybrid DRS-US probe may be a handheld probe. Alternatively, its placement may be controlled by a robot.

[0114] In certain embodiments the measurement device 901 is not hand-held. The measurement device may be incorporated into a larger, stationary machine that can perform analysis of a sample.

[0115] Ground truth labels may be provided to train a learned model to interpret the multi-depth DRS data. Assignment of ground truth labels for the multi-depth DRS dataset may be based on combined information extracted from corresponding US images and histopathology results. For example, the US images may be manually annotated to generate the ground truth labels, as opposed to production environments where the US images may be processed automatically in certain embodiments to find e.g. locations of tissue layer transitions.

[0116] In certain advantageous embodiments, DRS spectra from multiple sampling depths can be analyzed together in one tumor classification model. For example, such a tumor classification model can be trained and / or configured to predict the presence of any tumor tissue within each of a plurality of depths from the resection surface that has been examined by the DRS. This way, DRS may be used for margin assessment without any involvement of ultrasound. Alternatively, the DRS data may be advantageously combined with ultrasound data for improved results, as elaborated herein.

[0117] In certain embodiments, uncertainties in the ground truth tumor margins and DRS measurement volume may be included as annotations of the ground truth values, for improved training of a learned model.

[0118] Fig. 1A shows a perspective illustration of a hand-held optical probe 99. Fig. 1 B shows a view of the outside surface of the probe, where the optic fibers end, so that they can perform light transmission of light out of the tip of the optic fiber and reception of light into the tip of the optic fiber.

[0119] Fig. 1 B in particular shows the tips of the optic fibers. The optic fibers are aligned in a line, although this is not a limitation. The optical fibers may have a core diameter of, for example, 400 pm, depending on the application. The optic fibers could be arranged in any pattern. The distance between detecting fiber and emitting fiber is important, as it is related to the depth of the measurement. The arrangement of Fig. 1 B shows one detecting fiber 110 and six emitting fibers 101-108. The distance between detecting fiber 110 and emitting fiber 101 is 1 millimeter. The distance between detecting fiber 110 and emitting fiber 101 is 1 millimeter. The distance between detecting fiber 110 and emitting fiber 102 is 2 millimeter. The distance between detecting fiber 110 and emitting fiber 103 is 3 millimeter. The distance between detecting fiber 110 and emitting fiber 104 is 4 millimeter. The distance between detecting fiber 110 and emitting fiber 106 is 6 millimeter. The distance between detecting fiber 110 and emitting fiber 108 is 8 millimeter. Other distances may be considered in addition to, or alternatively to the distances shown here. In many applications distances in the range of 1 millimeter to 10 millimeters are preferred. In certain embodiments, the roles of detecting fiber and emitting fiber may be reversed. For example, there may be one emitting fiber at the location shown as 110, and there may be detecting fibers at the locations shown as 101- 108.

[0120] Fig. 1C shows a sketch of possible radiation deflection paths 123 from each of the emitting fibers 101-108 through a tissue sample 124 to the detecting fiber 110. These deflection paths 123 could also work the opposite way, from an emitting fiber at location shown 110 to a detecting fiber at any one of the locations shown at 101-108, for example. As shown, the further apart the emitting fiber and the detecting fiber, the further into the tissue the light penetrates before being deflected into the detecting fiber.

[0121] Fig. 1 D shows another arrangement of the fiber tips, in which a plurality of emitting fibers 151 and detecting fibers 152 are interleaved. In Fig. 1 D the emitting fibers 151 and the detecting fibers 152 are arranged alternatingly in a row, forming a row of pairs of an emitting fiber 151 and a detecting fiber 152. The distance between each pair of an emitting fiber 151 and a detecting fiber 152 may be the same for all pairs. For example, the emitting fiber to detecting fiber distance may be in a range of 0.5 to 10 mm, preferably in a range of 1 mm to 3 mm, more preferably about 2 mm. As illustrated in Fig. 1 D the distance between the fiber tips of a pair may be smaller than the distance between any two fiber tips of two different pairs. However, this is not a limitation. Also, the distance between neighboring fibers may vary to measure different sampling depths. Moreover, the fibers may be arranged in multiple rows, for example multiple parallel rows, or in any other planar arrangement.

[0122] Fig. 1 E shows a sketch of possible radiation deflection paths 153 from each emitting fiber 151 to the corresponding detecting fiber 152 of each pair of emitting fiber 151 and detecting fiber 152 through the tissue sample 154. As shown, together the adjacent pairs of fibers cover a larger second region of interest than the configuration of Fig. 1C. The same fiber configuration also can be used for measurements at multiple measurement depths. For example, activating a first emitter 151 , the light signal can be received by the neighboring detecting fiber(s) 152, but also by the other detecting fiber(s). For example, if the distance between the fibers is 2 mm, the signal can be detected by the detecting fiber at 2 mm away from the emitting fiber and by the detecting fiber at 6 mm away from the emitting fiber. By adaptively switching the fiber at 4 mm away from the emitting fiber to optically connect to a light sensor when not in use as an emitter, that fiber can also be used as a detecting fiber to provide a measurement at 4 mm away from the emitting fiber. Interleaved arrangement of emitting fibers and detecting fibers may also entail different arrangements, such as, two or more neighboring detecting fibers in between every two emitting fibers. Alternatively, two or more neighboring emitting fibers may be arranged in between every two detecting fibers. For example, the distance between neighboring fibers may be 1 mm or variable. This way, emitting fiber to detecting fiber distances may be realized to cover a suitable range of sampling depths, such as 1 , 2, 3, 4, 6, and 8 mm. The further apart the emitting fiber and the detecting fiber, the further into the tissue the light penetrates before being deflected into the corresponding detecting fiber.

[0123] An example of an optical scanner is a diffuse reflectance spectroscopy system. Such a system may comprise a number of broadband halogen light sources. An example of such a light source is manufactured by Avantes, AvaLight-HAL, 360 - 2500 nm. For example, each light source is configured so that it emits light into one of the emitting fibers 101-108, to be able to emit light into the tissue with a plurality of different emitter-to-detector distances.

[0124] In another example, the system may comprise one or multiple light-emitting diode (LED) light sources, where each LED may generate light at a different wavelength.

[0125] In another example, the system may comprise one light source connected to a fiber-switch or fiber-multiplexer, which is controlled by a control unit to allow light to pass from the light source to each emitting fiber in turns. For example a fiber-switch 1x4 can switch the path between one light source to four different emitting fibers. The light sensor may be likewise be coupled to all the detecting fibers, with or without a switch. Alternatively, the light source may be directly coupled to all emitting fibers, so that all emitting fibers emit the light simultaneously. The detecting fibers may be switched to different light sensor, e.g. photosensitive components, or may be connected to the same photosensitive component via a 1x4 switch to interpret the signal on each detecting fiber sequentially.

[0126] In addition, the system may comprise one or more spectrometers, or photodiodes. For example, multiple spectrometers may be provided to detect different wavelength ranges. For example, a spectrometer may be provided that covers the visible wavelength range (For example, Avantes, AVASPEC-HS2048XL-EVO, 200 - 1160 nm) and another spectrometer may be provided that covers the near-infrared wavelength range (Avantes, AVASPECNIR256-1.7-RS, 900 - 1750 nm). These different spectrometers may be optically coupled to the same detecting fiber 110. In certain embodiments, the light source comprises an LED and the light sensor comprises a photodiode and / or a photomultiplier. In certain embodiments, the light source comprises a broadband light source, such as a halogen light, and the light sensor comprises a spectrometer.

[0127] In cases of fiber-arrays with multiple light emitter fibers and multiple detector fibers, the system may comprise one or more switches or fiber-multiplexers for receiving fibers. For example, a fiber-switch 4 by 1 , can switch the path between 4 different receiving fibers and a single spectrometer. In cases with use of two spectrometers, one for visible range and one for NIR range, two fiber-switches or multiplexers may be used to couple the incoming signal from one of the detecting fibers to both the spectrometers.

[0128] A control unit, which may comprise a computer processor and memory with instructions to program the computer processor, may be configured to control the light source(s) and light sensor(s), such as spectrometer(s), and save and / or process the acquired data. Measurements may be performed using such a fiber-array optic probe, consisting of multiple optical fibers. Some fibers may be used to transport the light from the light sources to the tissue, at multiple different distances from the receiving fiber. Fig. 1C shows light paths from each of emitting fibers 101-108 to detecting fiber 110. Since the sampling depth of DRS depends on the distance between the sending and receiving fibers, the use of multiple fiber distances makes it possible to retrieve information from multiple sampling depths, as illustrated in Fig. 1C. The receiving fiber(s) may be used to transport the reflected light back from the tissue to the spectrometers.

[0129] A portable ultrasound scanner may be configured to acquire the ultrasound data as described herein. For example, the portable TELEMED Micrlls Ext-1 H device could be used as a beamforming example device, to acquire ultrasound device. Alternatively, the TELEMED L15-6L25S-3 transducer can be employed. The optical scanner may be built-into the ultrasound scanner or may be attached to the ultrasound scanner by means of a dedicated clip-on.

[0130] In certain embodiments, a hybrid DRS-US probe is created that can acquire coregistered DRS spectra and US images simultaneously using one handheld probe. This helps minimizing any differences in probe placement and applied pressure between the DRS and US measurements of the same location.

[0131] Fig. 2 illustrates that the optical fiber-array 1001 can be integrated with an ultrasound transducer 1000 to form a hybrid DRS-US device 220, as shown in Fig. 2A, or incorporated in a clip-on 222, which can be attached to an ultrasound probe 221 to form a hybrid DRS-US probe 220, as shown in Fig. 2C. In this exemplary DRS clip-on 222, the fibers enter on the side, adjacent to the US probe and have the tips of the fibers aligned with the ultrasound probe’s sensors. The exact same fiber distances for the tips of the fibers 210, 201-208 may be used as in the examples described and illustrated with reference to Fig. 1. The tips of the fibers may be arranged substantially parallel to each other and substantially parallel to the US imaging plane. The emitted light (as in Fig. 1C) parallel and overlapping the US imaging plane. Fig. 2A shows a schematic illustration of the fiber-array DRS probe integrated with the ultrasound probe to form a hybrid device 220, with, in this example, six emitting and one detecting fiber located next to the US imaging plane. Fig. 2B shows a view of the measurement surface of the clip- on 222 including the optical probe 1001 and a slot 223 for passing signals of an ultrasound (or other imaging) probe. Fig. 2C shows a perspective drawing of the fiberarray DRS clip-on 222 and ultrasound transducer 221 during measurement on a freshly excised colorectal specimen 225. Fig. 2D shows exemplary spectrograms 151-158 that could be detected for each of the six exemplary light emitter to light detector distances of light emitting fibers 101-108. The horizontal axis shows wavelength in nanometers (nm), the vertical axis shows reflectance in an arbitrary scale.

[0132] Fig. 2A also shows examples of the first region of interest 251 and the second region of interest 252. Such examples are also given in e.g. Fig. 12.

[0133] For example, DRS and US data may be acquired simultaneously using the hybrid probe. For example, a mold may be used to place ink marks on the specimen on the exact location of the DRS measurements. After the data acquisition at one location, the mold may be kept in place on the specimen to mark the location with ink.

[0134] Accurate labeling may be important for the validation of the DRS measurements and achieving good tissue classification performance. Although histopathology is frequently used as the "golden standard" when assigning labels to DRS data, the histopathology process (fixation and sectioning) can cause significant tissue deformation, resulting in a mismatch between distances measured during DRS data acquisition and the histology results (e.g. the resulting hematoxylin and eosin (H&E) sections. The term H&E may be used hereinafter to refer to histology analysis). This may lead to incorrect DRS label assignments and ultimately influencing the tissue classification performance when using such data to train a classification model. To address this issue, structural information from the corresponding ultrasound images obtained at the time of the DRS measurements, in addition to histopathology, may be used to improve quality of the ground truth data. This may improve preservation of spatial correlation and eliminate the time gap and processing steps associated with histopathology. In this ground truth extraction approach, the histopathology results may be used for tissue type characterization, while ultrasound imaging may be used to measure the tumor margins (distance between the resection surface of the specimen and the tumor boundary).

[0135] Fig. 3A illustrates a workflow for tumor margin determination to create ground truth data for tumor margin and / or tissue classification, including correlation with histopathology, annotation of the ultrasound images and the final ground truth tumor margin measurements.

[0136] The workflow of histopathology correlation, ultrasound annotations, and ground truth tumor margin extraction is summarized in Fig. 3A. During histopathological processing, the specimen is sliced (step 301) at the locations of the ink marks, after which the tumor is delineated (step 302) by a pathologist on the digitized tissue slices. Based on these histopathological results and the location of the ink marks, the tumor is delineated (303) in the acquired US image by an expert. Further, based on this delineation, for each DRS measurement location in the corresponding annotated US image, the ground truth tumor margin (in millimeters) is calculated (step 304). The extracted ground truth tumor margins can later be used for assigning ground truth labels to all six DRS spectra individually, as will be described elsewhere in the present disclosure. Also, an alternative technique to calculate the tumor margin by a learned model with a dedicated margin assessment head architecture will be described elsewhere in the present disclosure. In any optical scanner system used herein, a calibration may be useful to improve the data by correcting for system sensitivity and ambient light.

[0137] In the following, an example of a method for training a model to classify fiberarray DRS measurements is described. First, the process of assigning ground truth labels to the individual DRS spectra (observations) of all available fibers (e.g. six fibers) is defined. Subsequently, training a baseline tissue classification model is described. Furthermore, to account for uncertainties in the ground truth tumor margins and DRS measurement volume, the labeling strategy may be optimized using a grid-search approach by varying the width, depth, and scale factors and introducing an uncertainty region. Finally, the results may be integrated into a final classification model.

[0138] For example, as described above, we may have extracted the golden-standard tumor margins for all measurement locations, which may be used for assigning tissue type labels (e.g. healthy / tumor) to the individual DRS spectra of all six fibers. A conventional way of assigning labels to DRS measurements is based on the tissue composition within the sampling depth. The labels may be extracted by comparing a sampling depth threshold with the gold-standard distance to the tumor (tumor margin). The sampling depth is highly dependent on the emitter-to-detector distance (SDD) of the corresponding fibers, which is typically defined as approximately equal to the SDD of the fibers used. Subsequently, for each measurement location can be assessed whether the tumor lies within this distance from the tissue surface, based on the golden standard modality (e.g. pathology or US). If the tumor margin is smaller than the SDD, the corresponding spectrum may be labeled as Tumor. Conversely, if the tumor margin is larger than the SDD, the spectrum may be labeled as Healthy. This labeling process is illustrated in the upper part of Fig. 3B. These extracted labels may be used to train a baseline machine learning model. Each individual DRS spectrum may be considered as a separate observation, resulting in six observations per measurement location.

[0139] Different types of classification models may be trained using the described ground truth data. For example, examples of suitable models include linear support vector machine (SVM), a polynomial SVM, a K-Nearest Neighbors classifier, a Gaussian Naive Bayes classifier, a Random Forest classifier, a Gradient Boosting classifier, an extreme Gradient Boosting (XGBoost) classifier [T. Chen and C. Guestrin, “Xgboost: A scalable tree boosting system,” in Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, (2016), pp. 785-794], a categorical Boosting (CatBoost) classifier [A. V. Dorogush, V. Ershov, and A. Gulin, “Catboost: gradient boosting with categorical features support,” arXiv preprint arXiv: 1810.11363 (2018)], a Multilayer Perceptron classifier (MLP), a Light Gradient-Boosting Machine (LightGBM) classifier [G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu, “Lightgbm: A highly efficient gradient boosting decision tree,” Adv. neural information processing systems 30 (2017)], and an Adaptive Boosting (AdaBoost) classifier [Y. Freund and R. E. Schapire, “A decision-theoretic generalization of on-line learning and an application to boosting,” J. computer system sciences 55, 119-139 (1997)], or artificial neural networks. The performance may be evaluated, for example, using the average Matthews Correlation Coefficient (MCC), Area Under the Curve (AUC), accuracy, sensitivity, and specificity over 5 iterations.

[0140] It should be noted that the sampling volume (i.e. the shape and penetrating depth of the optical paths shown in Fig. 1C) of a DRS fiber is a complex function that depends on multiple factors such as the fiber diameter, the refractive index of the tissue, and the distance between the fiber and the tissue surface. Precisely characterizing the sampling volume of each DRS fiber is therefore extremely difficult. To label each DRS fiber observation as tumor or healthy in the baseline model, certain assumptions may be made. These include an arbitrary 5-pixel width (equivalent to 0.3 mm) in the US image to calculate the ground truth tumor margin, and a tumor margin threshold based on a rule of thumb that equates the sampling depth of DRS to the SDD. However, strict adherence to these assumptions could introduce a systematic bias in the labeling process and potential dataset corruption, especially when labeling observations with a ground truth tumor margin close to the sampling depth of the DRS fiber. Furthermore, it is important to acknowledge that the tumor margins extracted from the US masks are subject to small uncertainties as well and lack information in the dimension perpendicular to the US image. As a result, suboptimal labels may have been assigned during the initial labeling process, leading to misclassification and performance degradation of the baseline model.

[0141] To address the inaccuracy in DRS label assignments, several parameters may be introduced. For example, one or more of the following four parameters may be used. These parameters include a bias, the depth uncertainty margin, the tumor margin width, and a scaling factor, which are illustrated in Fig. 3B, the four bottom diagrams, from left to right: a diagram for bias = -0.5 mm, a diagram for uncertainty margin = 0.5 mm, a diagram for width = 3 mm, and a diagram for scaling factor = 0.5.

[0142] • Bias (b): may relate to the upward or downward shift of the expected emitted light penetration depth or measurement depth or ‘DRS sampling depth’ (wherein the ‘expected’ DRS sampling depth is, for example, set to be equal to the SDD, or set to another estimate of the light penetration depth, as desired, depending on the distance of the light emitter to the light detector). A negative bias was defined as moving the approximate sampling depth more superficial, while a positive bias means moving the sampling depth deeper into the tissue. Alternatively the bias may be replaced by the actual measurement depth.

[0143] • Depth uncertainty margin (u): when the ground truth distance to the tumor falls within this margin around the approximate DRS sampling depth (e.g. assumed to be equal to the SDD), the labeling was considered inconclusive and the measurement was excluded from the dataset. • Width (w): the horizontal extent in the US image, centered around the DRS measurement location, that was used to calculate the average distance to the tumor as ground truth.

[0144] • Scaling factor ( ): scaling the width and depth uncertainty margin based on fiber distances (SSD) for each individual fiber, according to the following equations: wscaied = f x SDD xwand uScaied = f x SDD x u. This parameter creates larger sampling volumes for larger SDDs.

[0145] In the first step, one or more types of classification model may be trained using the initially extracted labels as explained hereinabove. The best-performing classifier with the highest AUG value may be selected as a baseline model, which may be used for the label parameter optimization in the next step.

[0146] A grid-based optimization approach may be utilized to find the best parameters for assigning correct labels to the DRS observations. The ground truth dataset may be relabeled for various combinations of values of the four described parameters, after which the selected baseline model may be retrained. The effectiveness of the labeling strategy for each combination of parameters may be assessed based on the MCC of the resulting model, as the relabeling and introduction of an uncertainty margin could change the label distribution and class imbalance during testing. The same approach as described herein may be used for training and evaluation of the model.

[0147] Fig. 3B illustrates US-based tumor mask (zoomed-in, black-and-white colormap, white is tumor and black is healthy tissue) of an example measurement location with a tumor margin of 3.4 mm. The top image shows the baseline labeling process for two of the DRS fiber observations with SDDs of 1 mm (311) and 4 mm (312), which are labeled as Healthy and Tumor, respectively, because the latter 312 overlaps the white tumor region. The images in the bottom row show, from left to right, the effect of adding a bias 301 , adding an uncertainty margin 302, varying the width 303, and adding a scaling factor 304 on the labeling process. The bias 301 represents an up / downward shift of the expected DRS sampling depth (assumed to be equal to the SDD). The depth uncertainty margin considers the label of a measurement inconclusive (NaN) when the ground truth distance to the tumor falls within a certain margin around the expected DRS sampling depth, and excludes the measurement from the dataset. The width 303 represents the horizontal extent in the US image, centered around the DRS measurement location, that was used to calculate the average distance to the tumor as ground truth. The scaling factor 304 scales the width and depth uncertainty margin for each of the six fibers based on the SDD.

[0148] To mitigate the impact of random data split variability, instead of selecting the best-performing variable combination, the top 5% of the best-performing parameter combinations (or any suitable number of best-performing parameter combinations) may be identified based on the MCC value. The parameter values that occurred most frequently within this subset of best-performing combinations, while also considering the distribution of the neighboring values to avoid the selection of outliers, may be selected for the final model.

[0149] After identifying the combination of parameter values that resulted in the optimal labeling strategy, the performance of the different classifiers may be re-evaluated based on the new labels, and the best-performing classifier may be selected as the final classification model.

[0150] Fig. 4 shows a diagram of a process of training a model to analyze measurement data. The process is particularly suited for cases in which there is uncertainty as to a systematic error in the ground truth data. The method starts in step 401 by obtaining a baseline model. For example, this entails generating ground truth data for the available measurements in the training dataset. This ground truth data may be generated based on an assumption (e.g. an educated guess) for the values of certain unknown parameters. In case of classification of DRS datasets, this could include assumptions for the measurement depth of each DRS measurement and the ground truth tumor margin. The baseline model may be generated by training a learned model using the training dataset including the measurement data and the generated ground truth data.

[0151] In step 402, a set of variable parameters may be determined. These may be predetermined or selected based on a statistical analysis of the training data, for example. In case of DRS measurements, such parameters may include one or more of measurement depth bias, uncertainty margin, width, and scale factor, as explained above.

[0152] In step 403, various combinations of the values for the parameters are selected. This can be done by an educated guess or statistical analysis or domain knowledge.

[0153] In step 404, the ground truth classifications of the training data may be recalculated, for each selected combination of values of the parameters. In case of tumor detection by optical measurements, since e.g. sampling depth influences whether the light interacts with the tumor, if the parameter is related to sampling depth, the selected sampling depth may be compared to the ground truth tumor margin to set the ground truth classification of the optical measurement. This way, multiple different training sets with the same measurement data but different ground truth data may be created.

[0154] In step 405, a model is trained for each of the selected combinations of values of the parameters. In other words, a model is trained for each of training set corresponding to a selected combination of values of the parameters, as created in step 404. For example, the baseline model may be retrained based on the relevant created dataset. Alternatively, the models may be trained from scratch.

[0155] In step 406, performance is evaluated of the trained models according to a relevant performance criteria, such as percentage of correctly classified samples. One of the best performing models may be selected.

[0156] After that, the selected model may be incorporated into a scanner system (Fig. 11) as a software module, for example. Alternatively, the model may be applied as a cloud service to process datasets at a location remote of the actual scanner device. This model may be used in a clinical setting to assess tumor margins of excised specimen, for example, to aid the healthcare professionals to treat patients.

[0157] A study was performed on excised colorectal specimens from 106 patients. The study population consisted of 57 women and 49 men. Most of the tumors were located in the rectum (47%) and concerned an adenocarcinoma tumor type (80%). More than 50% of the patients received neoadjuvant therapy; chemotherapy, radiotherapy, immunotherapy, or a combination of these. Tumors of all stages were included, with four specimens showing complete pathological response to neoadjuvant therapy (pTO). The average tumor diameter was equal to 3.9 ± 2.2 cm.

[0158] The average margin to the tumor in this dataset was 5.73 ± 2.98 mm over all measurement locations, based on the manual annotations in the US images. For comparison, the tumor margin distances were extracted based on the pathology H&E sections as well, and compared to the margin found in US. A high mean absolute difference of 3.37 ± 2.79 mm was found between the two modalities, equal to ~60% of the average tumor margin in the dataset. These results confirm the previous findings in the literature regarding tissue deformation caused by the histopathological process [N. S. Goldstein, A. Soman, and J. Sacksner, “Disparate surgical margin lengths of colorectal resection specimens between in vivo and in vitro measurements: The effects of surgical resection and formalin fixation on organ shrinkage,” Am. J. Clin. Pathol. 111 , 349-351 (1999)] and our hypothesis that the pathology suffers from tissue deformations.

[0159] Fig. 5 shows evaluation results of eleven classifiers that were evaluated using the baseline labels as described hereinabove. Based on the results, the linear SVM classifier was selected as the best-performing baseline model, achieving an AUC of 0.86. This classifier was, therefore, used in the rest of this study for the label parameter optimization.

[0160] A grid-search optimization was conducted on the influence of four parameters during the label assignment, including bias, uncertainty margin, width, and scaling factor. The width and the uncertainty margin were varied between 0 and 2 mm, the bias between -0.5 and 0.5 mm, and the factor between 0 and 1. Initially, relatively large intervals in parameter values were chosen.

[0161] Fig. 6 shows the resulting classification performances. Again, diagrams from left to right show bias in mm versus MCC, uncertainty margin in mm vs. MCC, width in mm against MCC, and scale factor vs. MCC. The uncertainty margin had the most positive influence on the performance, whereas the introduction of the scaling factor negatively influenced the performance. The effect of the width and bias on the performance was less apparent, for the initially selected parameter intervals and ranges.

[0162] Subsequently, the scaling factor parameter was omitted from the analysis as it did not contribute to any performance improvement. Furthermore, based on the initial results as shown in Fig. 6, the parameter ranges and intervals for the numerical analysis were modified. The width was varied between 0 and 1.4 mm (intervals of 0.2 mm), the uncertainty margin between 0 and 2.4 mm (intervals of 0.15 mm), and the bias between -1 .5 and 1.5 mm (intervals of 0.2 mm). The larger the uncertainty margin, the higher the performance of the classification model, suggesting more accurate labeling. In combination with the use of a slightly negative bias during labeling, the model achieved the most optimal performance. The width for calculating the ground truth tumor margin had hardly any influence on the accuracy of the labels and classification performance.

[0163] After identifying the top 5% of the best-performing parameter combinations, based on the MCC value, histograms were created visualizing the frequency of each parameter’s value for these best-performing combinations. These histograms are shown in Fig. 7. Based on the results, a bias of -0.7 mm, an uncertainty margin of 2.4 mm, and a width of 0.2 mm were selected as optimal labeling strategy and used for training the final model.

[0164] Fig. 7 consists of three histograms showing, from left to right, the frequency of bias, uncertainty margin, and width values among the top 5% of best-performing parameter combinations.

[0165] Using the identified optimal labeling strategy, the performances of all classifiers were evaluated again, as shown in Fig. 8.

[0166] Fig. 8 shows performance metrics for different classifiers using the optimal labeling strategy. Optimizing the labeling strategy resulted in an increase in performance for all classifiers compared to the baseline model using the conventional labeling strategy. For example, the AUG ranged between 0.76 and 0.86 using the initial labeling strategy, while the optimal labeling strategy achieved an AUG ranging between 0.88 and 0.95.

[0167] The best-performing classifier (linear SVM) was able to distinguish tumor tissue from healthy tissue at various depths from the resection surface with an AUG of 0.95 ± 0.02, an MCG of 0.65 ± 0.05, and an accuracy, specificity and sensitivity of 0.92 ± 0.01 , 0.93 ± 0.01 , and 0.83 ± 0.06, respectively.

[0168] A fiber-array DRS probe with a plurality of emitter-to-detector distances may be provided for tissue discrimination at multiple depths. The optical fibers may be incorporated in a clip-on around an ultrasound transducer, to enable simultaneous acquisition of co-registered US images. These images may be used to obtain more accurate ground truth distances to the tumor at the time of the DRS measurements. This may avoid at least tissue deformation during the histopathology process, which often resulted in a mismatch between distances measured during DRS data acquisition and the resulting H&E sections.

[0169] One classification model may be trained using the DRS spectra from all sampling depths to predict the presence of any tumor tissue within various depths from the resection surface. A grid search optimization of the labeling process may be conducted to address any uncertainties in the DRS measurement volume and ground truth tumor margins, by varying several parameters (such as bias, uncertainty margin, width, and scaling factor).

[0170] One of the conventional DRS labeling strategies is to label a spectrum as Tumor when the ground truth distance to the tumor at the measured location is smaller than the SDD of the fibers used, assuming that the sampling depth of DRS is approximately equal to the SDD. By applying this strategy on our fiber-array DRS dataset, the tumor classification AUG ranged from 0.76 to 0.86 for all evaluated classifiers, indicating already a decent discriminatory ability. The linear SVM classifier achieved the best classification performance, with an AUG of 0.86, accuracy of 0.77, sensitivity of 0.63, specificity of 0.82 and MCG of 0.44 (see Table 2). However, the conventional labeling strategy is based on multiple assumptions regarding the expected DRS measurement volume. Therefore, a numerical analysis may be conducted to optimize the labeling strategy, aiming to improve the accuracy of the ground truth labels and thereby the trained classification model. Four parameters may be defined for the optimization process: the uncertainty margin, bias, width, and scaling factor. Other parameters may be defined to obtain further improvements.

[0171] Firstly, Fig. 5 and 6 showed that incorporating an uncertainty margin around the estimated sampling depth of each DRS fiber, and labeling observations as inconclusive when the ground truth distance to the tumor falls within this margin, led to improved performance. Excluding the observations with inconclusive labels during model development likely result in increased dataset reliability. The figures show an increasing trend in performance for larger uncertainty margins, indicating that increasing this margin further may lead to even better performance. However, increasing the depth uncertainty margin too extensively may reduce the added value of using the fiber-array DRS probe to measure at different depths, as it results in the exclusion of fiber observations.

[0172] Secondly, shifting the expected sampling depth of each DRS fiber more superficially may improve the performance of the baseline model as well, as could be seen in Fig. 6 and Fig. 7. A bias of -0.7 mm resulted overall in the best classification accuracy (Fig. 7). The results of this study show that the estimated measurement depth during labeling clearly influences the final classification performance.

[0173] Using the identified optimal labeling strategy, an improvement in performance was observed for all classifiers compared to the initial labeling strategy (see Fig. 8). Labeling optimization resulted in an average increase over all classifiers of 0.10 ± 0.01 for the AUG (12%), 0.13 ± 0.03 for the MCG (29%), 0.13 ± 0.02 for the accuracy (17%), 0.13 ± 0.03 for the specificity (17%), and 0.08 ± 0.06 for the sensitivity (12%). This indicates that the numerical analysis resulted in a more reliable dataset, rather than overfitting the dataset to the single classifier used as the baseline model (Linear SVM). The final model was able to distinguish tumor tissue from healthy tissue at various depths from the resection surface with an AUC of 0.95, an MCC of 0.65, and an accuracy, specificity and sensitivity of 0.92, 0.93, and 0.83, respectively.

[0174] Previous studies developed tissue classification models for DRS data acquired with a single SDD and mainly superficial. During colorectal surgery, multiple tissue layers will be encountered and it might be interesting for surgeons to receive guidance about the presence of tumor tissue at several distances from their probe. To address this limitation, the current study utilized multiple fibers with varying SDDs of, in the exemplary study, up to 8 mm. To the best of our knowledge, this is the first study in which a single classification model was developed for tumor detection at multiple depths from the resection surface, based on DRS spectra from multiple fiber distances. Such methodology for assigning labels to DRS measurements is applicable to different type of cancers such as breast, colorectal and sarcoma cancers.

[0175] Certain embodiments of the present disclosure combine diffuse reflectance spectroscopy (DRS) technology with ultrasound imaging. The ultrasound component may be able to give an overview image which enables the determination of where the tumor margin is at risk, while DRS may be able to determine the tissue types at that location, even in cases where such tissue type cannot as reliably be determined by ultrasound. For example, also in cases where the tumor shares a similar ultrasound echogenicity with surrounding tissue (e.g. fibrosis in colorectal cancer) or in the cases of small spot tumors (e.g. ductal carcinoma in situ in breast cancer), diffuse spreading (e.g. Myxofibrosarcoma) and in general in tumors with no well-defined borders. To this end, a combined device is disclosed herein to supply the surgeons with real-time visualization (imaging information) on the tissue structure and tissue type at the tip of the instrument, which enables them to optimally assess the surgical margin (up to several mm in depth) in the operating room.

[0176] Fig. 9 shows a functional block diagram of an example of a combined measurement apparatus 901. This measurement apparatus 901 may be constructed as one solid device in a housing, or as a plurality of devices that may be interconnected by a cable and / or wireless communication. Fig. 2C shows a perspective of a possible construction of such a combined DRS-LIS probe 220 on a specimen 225. However, other constructions are possible as well. The probe 901 has an imaging scanner 907 and an optical scanner 904. The probe may further comprise a control unit 902, with e.g. a computer processor, a memory, all configured to control the imaging scanner 907 and the optical scanner 904. Moreover, the imaging scanner 907 may be controlled by an optional imaging controller 910. For example, the imaging scanner 907 may comprise one or more transducers, and the imaging controller 910 may transmit and receive electronic signals to and from the transducer(s) of the imaging scanner 907, under control of the control unit 902. The control unit 902 may further control input / output of signals and / or data via communications port 903. Communications port 903 may comprise any data transmission technology such as WiFi, or any wired or wireless communication technology. The communication port 903 may be configured to transmit and / or receive operation commands, ultrasound data, optical data, and other data, under control of the control unit 902. The imaging scanner 907 may comprise an ultrasound transducer that generates and receives ultrasound signals, and the imaging controller may comprise an ultrasound backend. Alternatively, the imaging scanner 907 may comprise an optical coherence tomography imaging head, and the imaging controller 910 may comprise a backend for controlling the signals of the optical coherence tomography imaging head. Alternatively, another imaging device may be provided instead of, or in addition to, an ultrasound scanner. It will be understood that in certain applications the imaging controller 910 and the imaging scanner 907 may be integrated into one unit.

[0177] The optical scanner 904 may comprise any type of optical scanner, such as DRS, Raman, or autofluorescence (AF). The optical scanner 904 may comprise a light emitter 906, a light source 909, a light detector 905, and a light sensor 908. The optical scanner 904 may comprise a plurality of any or each of these four components.

[0178] The light detector 905 may be optically connected to the light sensor 908, for example by means of a light guide such as an optical fiber with optional optical switches. The light detector 905 may be configured to receive light from a sample external to the apparatus 901 and forward it to the light sensor 908. The light sensor 908 may comprise a light sensitive element such as a photodiode that converts the light into electric signals. The light detector 905 may comprise a distal tip of the optical fiber or light guide. In certain alternative embodiments, the light detector 905 may comprise a lens focusing the incoming light onto the light sensor 908 directly (without optical fiber). In certain embodiments, the light detector 905 may further comprise a lens or another optical component at the distal tip of the optic fiber, to direct the light into the optic fiber. On the proximal end of the optical fiber the light may be guided onto the light sensor 908. The light sensor 908 may comprise, for example, a spectrometer that generates electrical signals depending on the detected spectrum. The spectrometer may be a super dispersion micro spectrometer configured to detect a spectrum of light with wavelength bands suitable for, e.g., one or more of diffuse reflectance spectroscopy (DRS), photodiodes, auto-fluorescence spectroscopy (AFS), differential path length spectroscopy, and Raman spectroscopy (including the following sub classifications or variations SERS, SORS and INVERSE SORS). In a preferred embodiment, light sensor

[0179] 908 comprises a spectrometer configured to detect a DRS spectrum in a spectral band between about 300nm to 1700nm and preferably between about 300nm to 940nm.

[0180] The control unit 902 may be configured to control to perform a DRS analysis on the light measurement data to output DRS data, for example, based on the signal generated by the light sensor 908.

[0181] The optical scanner 904 may further comprise a light emitter 906 to emit a spectrum of light into a sample. The light emitter 906 may be configured to receive the light from the light source 909 through a light guide. The light source 909 may comprise any one or more of one or more light emitting diodes, one or more laser diodes, or a supercontinuum laser. The light emitter 906 may be optically connected to the light source 909 via a light guide, which may comprise optic fiber and optional optical switches. The light emitter 906 may comprise the distal tip of an optic fiber. The light emitter 906 may further comprise optional optical elements such as a lens. In certain alternative embodiments the optic fiber may be omitted, for example the light generated by the light source 909 may radiate light directly to an optic component such as a lens or transparent element of the light emitter 906. In certain embodiments the light source

[0182] 909 comprises a lamp that generates a broadband white light. In certain embodiments the light source 909 is configured to transform electric energy into light energy.

[0183] The sensitivity of the DRS measurement may depend on the light absorbance and scattering of e.g. a target biomarker as well as the specifications of the spectrometer and the light source. Control unit 902, connected to the optical scanner 904, is configured to process and / or retransmit the signals from the optical scanner 904. For example, the control unit may be configured to calculate a tumor margin or a tissue classification based on the received optical data and ultrasound data. Alternatively, the control unit 902 may be configured to transmit the raw data to another device via the communications port 903, so that certain calculations may be performed on another device. For example, the parameters may be calculated by the control unit 902 or external device using the methods set forth herein, in combination with known algorithms for one or more of diffuse reflectance spectroscopy (DRS), auto-fluorescence spectroscopy (AFS), differential path length spectroscopy, and Raman spectroscopy (including the following sub classifications or variations SERS, SORS and INVERSE SORS).

[0184] For example, the control unit 902 may be configured to control to operate the imaging scanner 907 and the optical scanner 904 to operate simultaneously, i.e. to emit and receive the optical signals by the optical scanner 904 simultaneously with performing an image acquisition with the imaging scanner 907. This control can be done via the optional light source 909, light sensor 908, and imaging controller 910.

[0185] The apparatus 901 may further comprise a display 1101 (illustrated in Fig. 11) configured to output the image generated using the image scanner 907 and an indication 1121 of the second region of interest with respect to the image. This allows a user of the device (or an artificial intelligence coupled to a robot) to identify a target region of interest in the image and reposition the apparatus to bring the second region of interest towards the target region of interest to make an optical measurement of the most relevant portion of the sample.

[0186] The optical scanner 904 may comprise a plurality of the light detectors 905 and / or a plurality of the light emitters 906, implemented in the way set forth, for example coupled to respective light sensor 908 and light source 909. In certain embodiments, the at least one light emitter 906 and the at least one light detector 908 form a plurality of light emitter to light detector distances. Pairs of emitter 906 and detector 905 with different distances in between them can be individually operated by the control unit 902, so that measurements may be made using each available light emitter to light detector distance. For example, the apparatus may be configured to sequentially or simultaneously activate different pairs of one light emitter of the at least one light emitter 906 and one light detector of the at least one light detector 905, wherein the different pairs are associated with different light emitter to light detector distances. Activation of a light emitter / light detector may be implemented by activating the corresponding light source 909 / light sensor 908 and optional optical switching. Preferably in certain clinical applications the light source to light emitter distances are in a range of 0-10 millimeters, preferably 1-8 millimeters. Different emitter to detector distances may be provided from a minimal distance to a maximal distance. These distances may be linearly distributed or (preferably) logarithmically distributed, for example. For example, the combinations include: a light emitter to light detector distance of at most 2 millimeters, e.g. in the range of 1-2 millimeters, a light emitter to light detector distance of at least 6 millimeters, e.g. in the range of 6-10 millimeters, preferabley 6-8 millimeters, and one or more light emitter to light detector distances (e.g. four different distances) distributed in the range from 2 millimeters to 6 millimeters.

[0187] In certain embodiments, the optical scanner 904 comprises at least 6 emitter to detector distances.

[0188] For example, the optical scanner 904, 1001 may be aligned in a center of the imaging scanner 907, 1000, as illustrated in Fig. 2A, so that the second region of interest 252 is registered to a middle of the first region of interest 251. The middle of the first region of interest may be measured along a contact surface of the imaging scanner 1000 with the sample 225 that is being examined. For example, the second region may extend from a line in the image representing the contact surface towards the center of the image. In other words, the second region of interest may extend from a boundary of the first region of interest, the boundary corresponding to the contact surface of the device 220, in particular the contact surface of the imaging scanner 1000 and / or the optical scanner 1001 , into the image. In certain embodiments, the second region of interest may be located in the middle of that boundary. The construction of the device 220 may be such that the relative position (including the orientation) of the second region of interest relative to the first region of interest is fixed. The exact position of the second region of interest relative to the first region of interest may be determined in advance (at the time of manufacturing or by means of a calibration procedure). The parameters defining this relative position may be referred to as a registration of the first region of interest with the second region of interest. These registration parameters may be stored in control unit 902, 1452. Similarly the optical clip-on 222 shown for example in Fig. 2C and Fig. 13 may be configured such that the optical scanner 1105 is at a well-defined fixed position relative to the imaging scanner when the clip-on is clipped on the suitable imaging scanner device 1106. Therefore, the first and second region of interest are similarly registered in the case of a clip-on embodiment. The control unit 902 may be configured to estimate a measurement depth of the optical scanner 904 based on the image. For example, by comparing images and optical measurements for a number of training cases, while studying the tissue transitions visible in the images and correlating this to the optical measurements, the measurement depth (i.e. the reach of the optical signal into the tissue before it is dispersed back to the light detector 905) may be estimated.

[0189] The control unit 902 may be configured to detect a transition from a first tissue layer with a first tissue type to a second tissue layer with a second tissue type. This may be done based on the image. For example, a tissue boundary may be identified using image processing techniques such as edge detection. Alternatively, the transition may be detected using the light measurement data. For example, by comparing the measurement data for two different light emitter to light detector distances, it may be determined that a tissue transition boundary lies between the sampling depths corresponding to the two different light emitter to light detector distances.

[0190] The apparatus may comprise a learned model configured to combine the image and the optical measurement data, which are registered to each other, to estimate an estimated tissue type classification and / or a depth estimation of a tissue type. The image and the optical measurement data may be combined to be able to better detect properties of the sample.

[0191] For example, the learned model may be configured to estimate, for a particular measurement comprising an image generated by the imaging scanner and an optical measurement by the optical scanner, a measurement depth associated with at least one light emitter to light detector distance of the optical scanner, based on the image.

[0192] In certain embodiments, the optical scanner 904 is implemented in a clip-on unit 222. The clip-on unit may be a separate device that can be removably clipped onto a housing that includes the imaging scanner 907. Mechanical means, such as snap-fit, may be provided to clip the optical scanner 904, detachably and at a reproduceable relative position, to the imaging scanner 907.

[0193] Certain embodiments do not include the optical scanner 904, the light sensor 908, or the light source 909. But they may comprise the imaging scanner 907 and imaging controller 910, control unit 902, and communication unit 903, as disclosed herein. Alternatively, certain embodiments may be implemented as a suitably programmed computer (not illustrated) configured to process image data generated by the imaging scanner 907 and imaging controller 910. For example, a medical system may be configured to calculate a distance from an image scanner 907 to a tissue layer in a sample, the system comprising a learned model 1500, which may be stored in a computer memory, for example. The learned model 1500 may be configured to process image data generated by an imaging scanner 907. For example, the learned model 1500 may comprise a backbone 1501 connected to a head architecture 1510 of a neural network. Further, a processor system 902 may be configured to estimate a distance from the imaging scanner 907 (or from another reference layer or reference line defined with respect to the image) to a tissue layer by applying the learned model 1500 to at least one image generated by the imaging scanner 907. The head architecture 1510 may be is configured to output a direct estimation of the distance from the reference layer (e.g. surface of the image scanner) to the tissue layer, based on a feature map generated by the backbone 1501. The reference layer or reference line may be substantially parallel to the transition boundary of the tissue layer. For example wherein the transition boundary is a boundary between a healthy tissue and a tumor tissue layer.

[0194] The head architecture 1510 may comprises a horizontal mean pooling layer 1506 configured to pool the feature map. This horizontal mean pooling layer 1506 may represent features of respective lines parallel to the reference layer or reference line. This may result in a feature map, for example a one-dimensional feature map. The head architecture 1510 may further comprise a one-dimensional convolution 1507 operating on the horizontal mean pooling layer. The one-dimensional convolution 1507 may comprise a number representing a likelihood of a tissue transition at a particular distance from the reference layer or reference line. A method may comprise receiving a medical image (for example a two-dimensional image), identifying the reference layer or reference line with respect to the image, and applying the learned model to the image.

[0195] Fig. 22 shows a method of using the apparatus 901. An imaging scanner 907 may be provided 2201 , aligned with an optical scanner 904, the optical scanner 904 comprising a light emitter 906 and a light detector 905. The image scanner 907 may be controlled to generate 2202 an image of a first region of interest using the imaging scanner. The light emitter 906 may be configured to emit 2203 light into a second region of interest using the light emitter 906 and e.g. light source 909. The light detector 905 may be used to detect 2204 the light that has interacted with a tissue in the second region of interest using the light detector 905 and e.g. light sensor 908. Before and during these measurement operations, the imaging scanner 904 may be aligned with the optical scanner at a fixed, well-defined location, so that the first region of interest overlaps the second region of interest, and the first region of interest is registered with the second region of interest. This registration may comprise known coordinates of the first region of interest relative to the second region of interest.

[0196] Fig. 23 illustrates a computer-implemented measurement method comprising: receiving 2301 an image of a first region of interest from an imaging scanner, receiving 2302 light measurement data corresponding to an optical measurement of a second region of interest, wherein the first region of interest overlaps the second region of interest, accessing 2303 parameters of the registration of the first region of interest with the second region of interest, and detecting 2304 a tissue type in the second region of interest based on the light measurement data and estimating 2305 a distance to the tissue type based on the image, and relating 2306 the detected tissue type to the estimated distance of the tissue type based on the parameters of the registration.

[0197] Fig. 24 illustrates a computer-implemented image analysis method. The method comprises receiving 2401 an image of a first region of interest, wherein the image is generated by an imaging scanner; and estimating 2402 a tumor margin using a learned model configured to process the image generated by the imaging scanner, the learned model comprising a backbone connected to a head architecture, wherein the head architecture outputs a direct estimation of the distance to the tissue layer, based on a feature map generated by the backbone.

[0198] Certain embodiments comprise a method of training a model to classify optical measurement data. This method may use image data measured with the imaging scanner 907 to improve the training method. The method may comprise a step of providing a training set comprising pairs of measurements, each pair comprising an image of a first region of interest of a sample and an optical measurement of a second region of interest of the sample, wherein the first region of interest is registered to the second region of interest and the first region of interest overlaps the second region of interest. The training method may also comprise a preliminary step of obtaining a baseline model 401. This baseline model may be a random initialization of a model. Alternatively, it may be a model that has received training, so that its outputs are already relatively good. The remainder of the method is to further improve this baseline model to perform better. Step 402 comprises determining at least one parameter representing an uncertainty regarding measurements stored in the training set. This could be a measurement depth. Also, initial measurement depth may be set based on an assumption, such as that the sampling depth is approximately equal to the distance from the light emitter to the light detector. The uncertainty parameter could relate to a deviation of reality from this assumption. Step 403 comprises selecting a plurality of combinations of values of the at least one parameter. The parameters may be searched using any suitable multi-parameter optimization procedure to find the optimal result. Therefore, many different selections are possible in this step, depending on the search method used. In step 404, a classification label is assigned to the optical measurement of each pair for each selected combination of values of the at least one parameter, wherein the classification label is extracted from the image of each pair under the assumption of the respective selected combination of values of the at least one parameter. This classification label may be used as a temporary “ground truth” or target values when further training the baseline model, to see how well the learned model performs when being trained to reproduce these temporary “ground truth” classification labels. This is performed in step 405, which comprises training a model for each selected combination of values of the at least one parameter, using the classification labels corresponding to the respective selected combination of values of the at least one parameter as target output values for the model.

[0199] For example, the step of assigning 404 the classification label may comprise: estimating a distance of a tissue type based on the image; setting an assumed sampling depth of the optical measurement (based on a particular combination of the parameters); and comparing the depth of the tissue type in the image to the assumed sampling depth. If the sampling depth exceeds the depth of the tissue type, then the optical measurement may be considered to be affected by this tissue type, so that the classification label may be set accordingly.

[0200] The at least one parameter may comprise, for example, at least one of: a depth bias (301) that augments an estimated measurement depth of the first measurement modality, a depth uncertainty margin (302) used to exclude measurements where a difference between the estimated distance to the tissue type in the image and the estimated sampling depth is within the depth uncertainty margin; a width (303) of a subregion of the first region of interest that is used to calculate an average distance to the tissue type in the image; and a scaling factor (304) that scales the width and the depth uncertainty margin.

[0201] As observed, for example in case of using the depth uncertainty margin 302 as a parameter, individual measurements may be excluded from the training set based on the parameter, so that the training sets of different models trained in step 405 may differ in terms of population in the training set. On the other hand, the other listed example parameters typically augment the classification label rather than the inclusion / exclusion criteria.

[0202] It will be understood that any of the methods disclosed herein may be implemented by means of a suitable programmed computer system (in cooperation with the hardware measurement tools, where needed).

[0203] Fig. 10 shows a perspective view of an example of a device according to the present disclosure. In certain embodiments, the block diagram of Fig. 9 could be used to implement the device 100. The device 100, which may be a hand-held device, comprises a housing 600. The housing 600 may be compact and light enough to be held and manipulated with one hand and may have an ergonomic design, which facilitates the use of the hand-held device 100. Optional elements like buttons or sliders, may be easily manipulated with the same hand or may be manipulated with the other hand, depending on the user. For example, the housing may have a button to power the handheld device and / or to start detecting. The housing 600 comprises at least one transmitting fiber 611 and at least one receiving fiber 612. The at least one transmitting fiber 611 transmits light signals from an internal light source 440, at its proximal end, to the examining site, wherein a distal end of the at least one transmitting fiber 611 optically communicates with an outside of the housing 600 and emits the light from the light source 440 into the environment (e.g. a tissue).

[0204] Optionally, the at least one transmitting fiber 611 transmits light signal from an optical port optically connected to an external light source. The light source may comprise multiple internal and / or external light sources, as well as multiple optical ports are plausible to transmit light to multiple transmitting fibers. Alternatively, multiple internal and / or external light sources may be optically switched, split, or combined to transmit light to one or more transmitting fibers. The light source, either internal or external, may be one or combinations of a broadband light source (for example, tungsten-halogen or mercury lamp) and one or more narrow band light source (for example, laser, light-emitting diode or filtered broadband light).

[0205] The first optical fiber 611 conducts light signal to biological tissues of the examining site. The tissues absorb, reflect, or back-scatter the light signal and generate a reflectance light signal that is collected at a distal end of the at least one receiving fiber 612, where the receiving fiber 612 optically communicates with the outside of the housing 600. The reflectance light signal has a spectrum containing information about the optical properties and structure of the biological tissues. An optical sensor 410, optically connected to a proximal end of the at least one receiving fiber 612, may be a photodiode or a broadband light detector. The optical sensor 410 may be a light detector to detect a spectrum of the received light signal, for example a spectrometer that generates electrical signals depending on the detected spectrum. The spectrometer may be a super dispersion micro spectrometer and may be configured to detect a spectrum of light with wavelength bands suitable for, e.g., one or more of diffuse reflectance spectroscopy (DRS), auto-fluorescence spectroscopy (AFS), differential path length spectroscopy, and Raman spectroscopy (including the following sub classifications or variations SERS, SORS and INVERSE SORS). In a preferred embodiment, optical sensor 410 is a spectrometer configured to detect a DRS spectrum in a spectral band between about 300nm to 1700nm and preferably between about 300nm to 940nm. The sensitivity of the DRS measurement may depend on the light absorbance and scattering of the target biomarker as well as the specifications of the spectrometer and the light source.

[0206] A processor 420, connected to the optical sensor 410, is configured to control activation of the light source 440 and to generate a parameter based on the electrical signal generated by the optical sensor 410, wherein the parameter may be at least one physiological parameter that is indicative of the tissue state (e.g., pathological, or not pathological) and / or a tumor margin. For example, the parameter may be calculated by the processor using the techniques disclosed herein.

[0207] The hand-held device offers the advantage to detect and process the light reflected by the tissues at a short distance from the probe, and in case of multiple emitter-to- detector distances, the detected light may be reflected by the tissues at several different distances from the probe. The hand-held device 100 may comprise a wireless transmitter 430 to transmit the parameter or other parameters possibly generated by the sensor to an external console or display. The hand-held device may be combined with hardware for other type of measurements, for example, an ultrasound transducer 1000, as described herein, which improves the specificity of the hand-held device to analyze tissues at the examining site. Other imaging modality that may be employed instead of ultrasound, include optical coherence tomography or X-ray tomosynthesis, and also tumor localization techniques such as radioactive seed localizers (e.g. Gamma probe) and magnetic seed finders.

[0208] To improve its autonomy, the hand-held optical device 100 may generate an output signal that is perceptible by a human being, e.g., a visible or auditive signal, indicative of the parameter. The hand-held device may further comprise at least one battery 615 to power the electronics. The battery may be replaced by any source of electric energy, such as a large capacitor and / or an energy harvesting component such as a solar panel. Alternatively, the device may be powered by an external power source.

[0209] In certain embodiments, a hybrid DRS-US probe enables simultaneous acquisition of co-registered DRS spectra and US images using a single handheld probe, by either incorporating optical fibers in a clip-on for the ultrasound transducer, or integrating optical fiber in an ultrasound device. Such a fiber-array DRS clip -on can be used not only in combination with ultrasound but also with other imaging techniques such as OCT, and also tumor localization techniques such as radioactive seed localizers (Gamma probe), magnetic seed finders.

[0210] Certain embodiments comprise a model for analyzing both modalities using a specific deep learning network head architecture for direct tumor-margin distance prediction, as disclosed hereinafter.

[0211] Certain embodiments comprise a display to visualize e.g. surgical margin and accurate indication of positive margin.

[0212] Mathematical models like the diffusion approximation may aid in predicting measurement depth based on these optical properties and probe attributes. These models account for light's propagation and scattering within the medium. Advanced techniques like Monte Carlo simulations offer more precise estimations by simulating how individual photons interact with the sample. However, these models typically rest on assumptions, and real-world variations in environmental conditions and tissue properties can lead to discrepancies between estimated and actual measurement depths. In practice, the DRS measurement depth can be validated using histology results, however the deformation during the pathology process can result in a mismatch of acquired DRS data and measurement location and hence inaccuracy in estimation of measurement depth.

[0213] Certain embodiments comprise a hybrid device to estimate the measurement depth in diffuse reflectance spectroscopy (DRS). In certain embodiments, DRS measurements with co-registered ultrasound (US) images may be combined to accurately depict tissue structures using an automated Al model. By utilizing the tissue structure extracted from the ultrasound images, a more precise estimation of the measurement depth for each emitter-detector fiber distance within the fiber array may be performed. This advancement enhances the accuracy of tissue discrimination and margin assessment, yielding significant improvements in the overall effectiveness of the technique.

[0214] Fig. 11 shows another embodiment of a combined ultrasound-optical measurement system. The system comprises a console 1101 , in this case having the form of a tablet, connected to an ultrasound beamforming and optoelectronics module 1102. The module 1102 is connected to an ultrasound probe 1104 and to an optical scanner, e.g. a DRS probe 1105. The DRS probe 1105 is integrated with a clip-on 1106 that comprises a sleeve in which the ultrasound probe snap-fits, and a slot through which the ultrasound probe extends when snap-fitted in the sleeve so that the ultrasound probe can directly contact a tissue-to-be-measured. The module 1102 may contain a computer system that controls the ultrasound probe 1104 and the DRS clip-on 1106. The module 1102 may further comprise optical light source(s), spectrometer, photosensitive components, connected to the DRS probe 1105 via the optical fibers. The module 1102 may further comprise ultrasound beamforming and signal processing electronics. Some or all of such electrical components may alternatively be integrated in the probe 1104 / 1105. The hybrid device 1104-1107 may be used to examine a specimen 1108, as illustrated.

[0215] Fig. 12 shows a detail of the ultrasound probe 1104 and the optical scanner 1105 and the clip-on sleeve 1106. Fig. 13 shows a frontal view of the clip-on 1106 with the optical scanner 1105 and the slot 1107 for the ultrasound probe 1104. Also shown is an enlargement of the optical scanner 1105 with indication of the light emitter to light detector distances. Fig. 14 shows another block diagram of a combined optical / ultrasound system. Wherein the ultrasound system could be replaced by another imaging system. The system comprises a combined probe 1401 comprising the ultrasound probe 1402, which may comprise a plurality of ultrasound transducers, forming the imaging scanner, and the optical fibers 1403 forming the optical scanner. The combined probe 1401 may be formed by a single hand-held device. Alternatively, the combined probe 1401 may comprise the ultrasound scanner 1402 with the optical scanner 1403 removably attached thereto, for example by means of a clip-on.

[0216] The system further comprises a module 1451 , which could be a box containing the illustrated components. The module 1451 may be a separate device (or multiple connected devices) from the combined probe 1401 , as module 1102 in Fig. 11 , or may be integrated into the probe, as in Fig. 10. The components of the module 1451 may be connected by means of optical fiber (to the optical scanner) and electric cable (to the imaging scanner). Other kinds of cable may be used alternatively. The module 1451 may comprise a control unit 1452, for example a computer or processor, that controls the system and / or performs signal processing operations. The module 1451 further may comprise an ultrasound controller 1455, configured to generate the control signals for the ultrasound transducers of the ultrasound probe 1402. For example, the ultrasound controller 1455 may comprise an ultrasound beamformer or known ultrasound original equipment manufacturing (OEM) component, operatively connected to the ultrasound probe 1402. The module 1451 may further comprise a power supply 1454 e.g. a battery or a connection to a mains network. The module 1451 may further comprise a communication port 1453 for communication of signals with external servers and / or user interface devices such as display, keyboard, buttons, touch screen, and / or mouse. The module 1451 may further comprise an ultrasound controller 1455, which may comprise an ultrasound beamformer. The module 1451 may further comprise the optical components 1455 of the optical system. The optical components 1455 may comprise the light source(s) 1456, such as broadband light source and / or light emitting diodes (LEDs), light detector(s) 1457, such as spectrometer(s) and / or photodiode(s), and one or more switches and / or fiber multiplexers 1458, to selectively connect the light source(s) and light sensor(s) to the tips of the optical fibers of the optical scanner 1430. In alternative embodiments the ultrasound scanner 1402 may be replaced by another type of imaging scanner and the ultrasound controller 1455 may be replaced by a corresponding imaging controller that generates the signals for the imaging scanner.

[0217] Fig. 15A shows another perspective view of a combined ultrasound / optical scanner. Fig. 15B shows a part of the optical scanner comprising tips of optical fibers forming the light emitter(s) and light detector(s), and a slot for the ultrasound scanner. The ultrasound scanner itself is not shown in the figure. Fig. 15C shows a view of the light emitters and light detectors, which may be tips of optical fibers, and their distances. In the shown example, there is a distance of 2 mm between adjacent emitters / detectors, although this distance is only an example and by no means a limitation.

[0218] Fig 15D illustrates sampling depths of neighboring emitter / detector pairs of Fig. 15C. The light emitters and light detectors are alternatingly arranged in a row. However, other arrangements of the light emitters and detectors in e.g. a plane are also possible.

[0219] For example, there could be a plurality of light emitters in between every two light detectors. Alternatively, there could be a plurality of light detectors in between every two light emitters. In these two alternative arrangements, the larger measurement area, achieved by having multiple light detectors and multiple light emitters, can be combined with the measurements at different measurement depths, achieved by the plurality of distances achieved between pairs of light emitters and light detectors.

[0220] However, in certain embodiments, these different measurement depths can be achieved even if the light detectors and light emitters are arranged strictly alternatingly, as in Fig. 15D, by selectively activating the light emitters (not illustrated). Indeed, the light emitted by an active light emitter can be detected by any of the available light detectors in the optical scanner. So, by keeping any light emitters in between an active light emitter and an active light detector inactivated, different light emitter to light detector distances can be realized in the configuration of Fig. 15C / 15D, even if there is an inactive light emitter in between the active light emitter and an active light detector. This way different sampling depths can be realized.

[0221] Fig. 15E shows an example implementation of the optical components 1455 together with the optical scanner 1403. However, it should be understood that this implementation is only an example. In certain advantageous embodiments, the arrangement of Fig. 1 B and 1 C and 2B and 13 may be realized with switches to measure at different sampling depths. Another aspect comprises a novel network architecture for direct and accurate tumor margin estimation in images depicting a tumor.

[0222] Accurate resection margin assessment during oncological surgery is essential for the clinical outcome and patient prognosis. Certain embodiments comprise a network for direct and automatic tumor margin prediction in ultrasound images of freshly excised breast and colorectal cancer specimens. More in general, certain embodiments comprise a network for direct and automatic tumor margin prediction in an image that shows the tumor and surrounding tissue.

[0223] Certain embodiments comprise a network head architecture, which directly estimates the tumor margin (in millimeters, or in pixels, for example) based on the feature map output of a neural network structure, such as an II PerNet backbone.

[0224] During testing, a network developed using the techniques disclosed herein achieved an average tumor margin prediction accuracy of 0.64 mm over a combined breast and colorectal data set. Using a mean absolute percentage error (MAPE) loss. These results demonstrate a significant improvement over conventional segmentation based margin prediction approaches used in previous research on the same data sets. An accurate automatic resection margin assessment network may enhance surgical outcomes and minimize the need for additional interventions or re-operations in the future.

[0225] Ultrasound imaging may be used for intraoperative margin assessment. Ultrasound offers many advantages, including real-time imaging and deep tissue penetration. Furthermore, it is a quick, non-invasive, and highly available tool. However, the processing technique disclosed herein may be applied to other imaging modalities that can perform sufficient degree of tissue penetration. Studies have shown that ultrasonic wave propagation in tissue is dependent on histological features including tissue microstructure and tissue heterogeneity. Several studies have found satisfactory results regarding the use of ultrasound to differentiate between tumorous and healthy tissue during breast conserving surgeries. Despite this, intraoperative ultrasound has not been routinely used yet for intraoperative margin assessment due to the need for training and experience with ultrasound image interpretation.

[0226] Deep-learning-based methods could solve the issue of ultrasound image interpretation. A straightforward approach for tumor margin assessment is performing tumor segmentation. Convolutional neural networks (CNNs) can perform automatic tumor segmentation in ultrasound images of surgical specimens. Subsequently, the tumor-margin distance can be calculated based on these extracted tumor masks and compared to the tumor-margin distance observed by human experts. However, segmentation models are not optimized for margin assessment, since the surgical margins are calculated from the upper boundary of the tumor while the segmentation models are trained to provide the best performance on the whole tumor boundary. Due to acoustic shadowing, the lower boundaries of the tumor are more difficult to segment resulting in less accurate overall segmentation, which ultimately could hamper the margin estimation. The lower boundaries of the tumor can also not be segmented in cases where the tumor is too large to fit entirely in the field of view. In clinical practice, during surgery a surgeon is probably solely interested in the upper tumor boundary for the resection margin assessment. This redefines the task to predicting the closest margin from the tissue surface to the tumor, without needing to segment the full lesion.

[0227] Certain embodiments comprise a network architecture which directly outputs the margin and thereby aligns the training and inference objective. The architecture may build upon a neural network backbone, for example a deep learning neural network architecture, such as the UPerNet backbone. In certain embodiments, the backbone is supplemented with a new head architecture. In deep learning, the network head refers to the final layers of a neural network model that are responsible for producing the desired outputs or predictions. These layers may be connected to the last hidden layer of the network and are designed to transform the learned features into a final output format. The proposed head architecture is designed in such a way that it outputs a direct estimation of the tumor margin (in any unit, such as millimeters or pixels) with the feature maps generated by the backbone as input.

[0228] Therefore, certain embodiments comprise a deep learning model on direct and automatic tumor-margin distance prediction in ultrasound images of excised cancer specimens. Furthermore, certain embodiments comprise a model that predicts the tumor-margin distance real-time in ultrasound images of cancer specimens from multiple different solid organs.

[0229] Certain embodiments comprise a CNN model with a novel head architecture configured to output direct tumor-margin distance prediction in ultrasound images of cancer specimens. Certain embodiments comprise a model that has been trained and tested on the ultrasound images of at least two different types of cancer: for example, colorectal cancer and breast cancer.

[0230] Certain embodiments comprise a method of training the neural network and / or the head architecture. Ground truth data may be generated in the manner described hereinabove. After the acquisition of each ultrasound image for the training and testing dataset, the measured location on the specimen can be marked with black pathology ink. Subsequently, the specimens can be processed according to standard protocols at the pathology department, after which a separate tissue slice can be created for every inked location. In all corresponding digitized H&E sections, the tumor region can be annotated by a pathologist. Based on these results, the tumor boundaries can be manually delineated in every acquired ultrasound image by an expert. This may result in a pixel-level ground truth tumor mask for each US image.

[0231] Examples of a breast and colorectal ultrasound image are shown in Fig. 16, including the corresponding tumor delineations (dashed) based on manual annotations, and the tumor margin (vertical arrows). Fig. 16A shows a breast ultrasound image and Fig. 16B shows a colorectal ultrasound image.

[0232] The ground truth tumor margin may be calculated as the distance between the top of the tumor mask and the top of the ultrasound image (that is, the tissue surface) as illustrated with vertical white arrows in Fig. 16. This pixel distance may be converted to any suitable unit such as millimeters based on the original dimensions of the ultrasound image. Of course, the tissue surface may be identified elsewhere in the image, on any edge of the image or somewhere else within the image (depending on the imaging modality).

[0233] In certain embodiments, when the purpose is resection margin assessment close to the specimen surface, the ultrasound images may be cropped to the top square of the images containing at least part of the tumor and up to the tissue surface.

[0234] Conventional image segmentation consists of the task of assigning semantic labels from a predefined set of classes to each pixel. In tumor segmentation, each pixel may be either assigned a healthy or tumor label and the task becomes a binary classification problem on a pixel level. However, in a clinical setting, a surgeon might be solely interested in the resection margin. This redefines the task to predicting the closest margin from the specimen surface to the tumor, without needing to exactly segment the full tumor.

[0235] State-of-the-art semantic segmentation architectures like ll-net [Olaf Ronneberger, Philipp Fischer, and Thomas Brox. ll-net: Convolutional networks for biomedical image segmentation. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 9351 , pages 234-241. Springer Verlag, 2015] and UPerNet [Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified Perceptual Parsing for Scene Understanding. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 11209 LNCS, pages 432-448, 2018] output a probability distribution over all predefined classes for each pixel. This can function as a proxy to localize the full tumor, of which the top row can be interpreted as the margin. However, this proxy introduces a misalignment between the training and inference objective. Especially, the bottom of the tumor may be less important when predicting the margin, whereas it is equally important as the top of the tumor for semantic segmentation.

[0236] Certain embodiments, comprise a network architecture to directly output the margin. This way, the training and inference objectives may be aligned, which may lead to improved results. The architecture may build upon any learned model that can process the images and generate a feature map. For example, the architecture may build upon the UPerNet backbone. Certain embodiments build upon this learned model structure with a new head architecture. This head architecture may be referred to herein as the margin assessment head. The proposed network head may comprise a mean pooling layer on the width dimension, followed by 1 D convolutions, a softmax layer, and a center of gravity computation on the height dimension. The softmax layer and center of gravity computation may be replaced by any determination of a single value based on the one-dimensional feature vector, such as selecting the index of the largest feature value of the feature vector. This center of gravity (or other single value such as index) may be multiplied by a scalar converting the feature map dimension to the original ultrasound height dimension in millimeters. The head block may be designed in a way to output a direct estimation of the tumor margin with the feature maps (output of backbone) as inputs. Fig. 17 shows a visualization of an example of architecture including a backbone 1501 and a margin assessment head 1510. The backbone 1501 may comprise a II PerNet block and is connected to a feature pyramid network 1503. Block 1502 indicates a Pyramid Pooling Module (PPM) block that forwards and processes data from the backbone 1501 and forwards it to the feature pyramid network 1503. The output of the feature pyramid network 1503 is fed to a fuse block 1504, and the output of the fuse block 1504 is a fused feature map 1505. It will be understood that the fused feature map 1505 may alternatively be created by any suitable network architecture. The fused feature map 1505 may be processed by the head architecture 1510. Specifically, the fused feature map 1505 is processed by horizontal mean pooling 1506 to provide a two- dimensional feature map. The two-dimensional feature map is processed by a 1 D convolution 1507 to output a one-dimensional feature map. Finally, a softmax and / or center of gravity operation (collectively indicated by 1508) is performed on the onedimensional feature map to generate the estimate of the distance (e.g. the tumor margin).

[0237] Details of the example head architecture layers are as follows:

[0238] Horizontal mean pooling 1506: converts each 2D channel feature map to 1 D vectors by mean pooling over the width of each channel.

[0239] 1D convolution layer 1507: converts the channel vectors into one channel vector.

[0240] Softmax activation: function that turns the one channel vector into a normalized probability distribution.

[0241] Centre of gravity: estimates the tumor margin by multiplying the Softmax layer output with indices (representing vertical height in the image).

[0242] Certain embodiments calculate a Mean Squared Error (MSE) as a loss function to train a neural network, in which the loss for a prediction * given label equals (¥) - ¥?)2This loss function scales quadratically with the absolute error, making gradient updates sensitive to outliers. Since many smaller errors are desirable over one large error in a clinical setting, the MSE loss was chosen as the baseline in the current study. However, another loss function may be used in other implementations.

[0243] The MSE loss however, does not take into account the absolute value of the ground truth. Absolute errors of several millimeters are punished equally for shallow and deep margins. This is not desirable for the clinical application during oncological surgery, in which a tumor margin of <1 mm is generally considered as a positive surgical margin, which should be prevented. Errors for shallow margins are therefore more important, as they can make the difference between a positive and a negative margin, and thereby influence surgical decision-making. On the contrary, errors for deeper margins have less influence on surgical decisions.

[0244] Therefore, certain embodiments employ a loss function that incorporates the absolute value of the ground truth margin into the loss function, such as the Mean Absolute Percentage Error (MAPE), defined as:

[0245] The MAPE loss punishes errors proportionally to the magnitude of the margin. The loss function may be used to train the model, by using gradient descent or other optimization techniques to optimize the parameters of the model (in particular the backbone) in order to minimize the loss function for the cases in the training data.

[0246] Two metrics were used to evaluate the accuracy of the tumor margin prediction across all test set images: 1) the mean absolute error (MAE) and 2) the mean absolute percentage error (MAPE). The mean absolute tumor margin error measures the average vertical distance, in millimeters, between the top tumor pixel in the predicted mask and the top tumor pixel in the corresponding ground truth mask. For the mean absolute percentage error, the tumor margin error is divided by the true tumor margin for every image, expressed as a percentage. This measure thereby provides a relative error, since achieving smaller tumor margin prediction errors is most important for smaller tumor margins. For both performance metrics, a smaller value is indicating better performance of the network.

[0247] In the following, results of training embodiments of the backbone with the margin assessment head, will be described. Ultrasound images were acquired on specimens from 86 breast cancer patients and 74 colorectal cancer patients, to form a training set used to train the embodiments.

[0248] The resulting tumor margin prediction errors achieved in the described study are shown in Fig. 18, for the entire training data set as well as for the breast and colorectal subsets separately. For the entire training data set, the mean absolute error (MAE) was 0.64 mm and the MAPE was 20.7%. Fig. 19 shows predicted tumor margins versus true tumor margins, for breast cases (B) and colorectal cases (C). The horizontal axis shows true tumor margin in millimeters, and the vertical axis shows predicted tumor margin in millimeters. The dotted lines indicate an error margin of 1 millimeter, which is frequently used as a resection margin in breast and colorectal cancer surgery.

[0249] Examples of the tumor margin predictions for multiple breast and colorectal ultrasound images are visualized in Fig. 20. The figure shows on the top row breast images and on the bottom row colorectal images. The continuous line is the predicted tumor margin and the dashed line shows the ground truth tumor margin, using the described margin assessment head architecture and the MAPE loss function.

[0250] These examples show that high tumor margin prediction accuracies were achieved for different appearances of tumor and surrounding tissue, despite the presence of any artifacts or darkness in certain parts of the image.

[0251] Certain embodiments comprise a model for automatic tumor margin assessment in ultrasound images of solid organs (e.g. breast and colorectal). To this end, the model may comprise a margin assessment network head architecture that outputs a direct prediction of the tumor margin based on the feature maps from the backbone.

[0252] In the experiments that were performed with the network architecture, the margin assessment accuracy (in this study 0.64 mm) of the model including the described head architecture approached the resolution of the ultrasound images that are used for training, (in this study ±0.5 mm) and is within the 1 mm resection margin that is generally desired in oncological surgery. Fig. 19 showed that all errors larger than 1 mm (indicated by the dashed lines) were consistently predicted smaller than the true margin. This means that the network typically predicts on the safe side and avoids missing positive resection margins, which is preferable to the opposite scenario. Two large absolute errors were observed for true margins of 8 mm, which are less clinically relevant as they would not affect the decision between a positive and negative resection margin. After examining these outliers more closely, they could be attributed to difficult-to-interpret ultrasound images resulting in inaccuracies in the manual tumor delineations.

[0253] The images with examples of tumor margin predictions (Fig. 20) demonstrate the network’s effectiveness across diverse scenarios, including variations in brightness, boundary characteristics (sharply defined or more gradual), size, shape, the amount of ultrasound gel, and the presence of artifacts due to a loss of probe contact. The images on the right show examples of some more challenging ultrasound images with larger prediction errors. The colorectal image with an MAE of 1.81 mm contains, for example, a tumor with small extensions into the surrounding fat tissue. In this case, manually delineating the correct tumor border can be challenging. After a re-assessment of the image, it was concluded that the tumor extensions might actually run slightly more towards the resection surface than initially delineated.

[0254] Fig. 21 compares the results of this study (“M2”) to previous publication (“M1”) using segmentation-based margin prediction approaches on the same data sets: Dinusha Veluponnar, Lisanne L De Boer, Freija Geldof, Lynn-Jade S Jong, Marcos Da Silva Guimaraes, Marie-Jeanne T F D Vrancken Peeters, Frederieke Van Duijnhoven, Theo Ruers, and Behdad Dashtbozorg, Toward Intraoperative Margin Assessment Using a Deep Learning-Based Approach for Automatic Tumor Segmentation in Breast Lumpectomy Ultrasound Images, Cancers, 15:1652, 32023. The model with the margin assessment head architecture shows a clear improvement in accuracy compared to the segmentation-based approaches. Moreover, this study presents a single model that works for multiple organs, unlike the previous studies that employed organ-specific models.

[0255] While in the described evaluation study, the network architecture was trained and evaluated on breast and colorectal ultrasound images, the proposed methodology can be extended to other organs and imaging modalities, expanding its potential application. The method and system may also be employed for in vivo evaluation during surgery, by processing ultrasound images (or other modality images) of live ultrasound acquisitions. The network may achieve (near) real-time predictions, processing an ultrasound image in approximately 200 milliseconds, for example. This rapid computation time enables visualization of multiple frames per second in the operating room, facilitating intraoperative decision-making. Accurate automatic resection margin assessment may enhance surgical outcomes and minimize the need for additional interventions or re-operations in the future.

[0256] Certain embodiments comprise a learned model, such as an artificial neural network, for automatic tumor margin assessment in medical images, for example ultrasound images of both breast and colorectal cancer, or other tumors. The learned model may comprise a network head architecture was introduced, which outputs a direct estimation of the tumor margin (in millimeters) based on the feature map output of the backbone. In certain embodiments, the learned model is trained using a mean absolute percentage error (MAPE) loss function. In certain embodiments, the network may be pre-trained, for example on a publicly available (breast) ultrasound image data set from, for example, both benign and malignant tumors.

[0257] In certain embodiments, comprise a learned model that takes optical measurement data for a plurality of light emitter to light detector distances, and an ultrasound image (or other kind of medical image), as an input, and that outputs a tumor margin and / or tissue type classification for a certain depth, based on the combined optical and imaging data.

[0258] In certain embodiments, a combined model may be configured / trained as follows. The measurement depth estimation of each light emitter to light detector combination may be determined, as described hereinabove, for example by using the ultrasound data as a way to determine the depth of a certain tissue.

[0259] After the accurate measurement depth estimation, a classifier model (for example, a DRS model) may be developed to be able to correctly classify each DRS spectra individually as tumor or healthy. This classifier model may be, for example, an artificial neural network.

[0260] Using the classifier model, each spectra is classified, the classification algorithm provides a probability of being tumor for each spectrum (a value between 0 and 1). The higher the value, the higher chance of being tumor.

[0261] For example, using a fiber array with 6 different emitter to detector fiber distances results in 6 individual spectra and 6 probability values for classifications relating to 6 different measurement depths.

[0262] For a combined analysis, the probability values may be concatenated with the 1 D convolution features obtained using the 1 D convolution 1507 of the head architecture of the model used to process the image data (e.g. the ultrasound image).

[0263] This set of combined features (containing the probability values of the optical measurements and the 1 D feature vector of the image / ultrasound data) may then be used for training a new prediction model that outputs the tissue classification (tumor yes / no) and / or the distance to the detected tissue type (i.e. tumor margin). This combined model may result in higher accuracy in e.g. tumor margin prediction.

[0264] In conclusion, certain embodiments perform margin prediction only using the US without optical measurements, with the introduced network head 1501 as shown in Fig. 17. Certain other embodiments use both US and DRS combined, as described above, replacing the softmax+center of gravity step 1508 of the 1 D Convoluted feature vector by a step of concatenation of the probability values obtained from the optic measurements and an additional learned model that processes the concatenated inputs to output a tumor margin. The latter type of embodiments resulted in more accurate margin prediction.

[0265] For example, in certain embodiments, one may use the US network (with the introduced head architecture) which has been already trained and for every new image, the hidden layer information (the 1 D convolution feature vector) may be extracted. A second (classification) model may be designed to predict a probability value of being tumor for each DRS spectrum. It's noteworthy to emphasize that in this model, the ground truth data for training may be derived partially or completely from US information. This unique approach enabled us to refine the measurement depth estimation for DRS, consequently leading to more precise labeling and better classification results.

[0266] For example, separate models (or one combined model) may be trained to classify tissue based on the measured spectrum of every available light emitter to light detector distance, for example. And by evaluating optical measurements for each light emitter to light detector separately, in combination with the known sampling depth of each light emitter to light detector distance, it may be possible to establish the distance from the tumor boundary to the optical scanner. This way, a margin may be determined using DRS as a standalone measurement, without using ultrasound.

[0267] Alternatively, the ultrasound image may be used together with the optical measurement data to obtain more accurate tissue classification. To that end, an additional model may be trained to map the feature vector output by the ultrasound measurement processing model (model 1) and the feature vector output by the optical measurement processing model (model 2) as input for a new combined prediction model (model 3). The feature vector output by the optical measurement processing model (model 2), may be for example a probability for each measured sampling depth that the tissue at the relevant sampling depth contains tumor tissue. For the new prediction model 3, a random forest regression model may be used in certain embodiments, or for example another type of regression model or an artificial neural network. The input of the model 3 may be, for example, 70 features (64 dimensional feature vector generated by the head architecture and 6 feature values of corresponding to 6 optical measurements) and the output may be the estimated tumor margin (in millimeters, for example). For example, the basis for the ground truth of tumor margin to train this new prediction model (model 3) may be manually annotated ultrasound images supplemented by histology results. It should be mentioned that automatic detection of tumor margin is a difficult task and the ground truth may thus be based on annotations by expert radiologists with prior knowledge. Alternatively, other means to annotate the images may be envisaged. In alternative embodiments, models 1 , 2, and 3 may be integrated into one learned model, such as a deep neural network.

[0268] For example, in cases where the tumor shares a similar echogenicity with surrounding tissue (e.g. fibrosis in colorectal cancer) or in the cases of small spot tumors (e.g. ductal carcinoma in situ in breast cancer) that are hardly visible, diffuse spreading (e.g. Myxofibrosarcoma) and in general, in tumors with no well-defined borders, US imaging alone may not always be sufficient for surgical margin assessment. In such a case, the additional use of information from DRS, as described above, can improve the margin prediction.

[0269] In order to combine the data from both modalities in a combined tumor margin prediction model, we used post-processing features extracted from the outputs of previously developed models that were dedicated to each modality individually. These features were then used to train a new regression model for tumor margin estimation.

[0270] Fig. 30 illustrates a combined processing of optical measurement data and image data. Herein the image data may be ultrasound images or other kinds of images, as described throughout the present disclosure. A supervised classification model 3003 may be developed for fiber-array DRS data 3001 , as described hereinabove. This model may be designed to classify each of the six DRS measurements per location, measuring at a plurality of different depths, into either healthy tissue or tumor tissue. Instead of the binary classification, the model 3003 may be leveraged to output the probability of being tumor tissue for each DRS measurement depth, providing more valuable information for addressing the regression problem. As a result, multiple DRS probability features 3005 (in a range from 0 to 1) may be obtained per measurement location, representing the likelihood of being tumor at a plurality of different measurement depths. An example is shown in the top row of Figure 30.

[0271] US feature extraction may be carried out using a previously trained deep learning model 3004 designed to estimate the smallest distance to the tumor within US images 3002 of breast and colorectal cancer. The final layer of this model 3004, which may comprise an activation map 3006 output by the horizontal mean pooling layer 1506, may be used as a feature vector 3006. This activation map may represent a heatmap indicating the pixels in the US image 3002 to which the model 3004 assigns high attention for predicting the tumor margin. Within this activation map, the pixels along the vertical dimension may be extracted at the location of the DRS measurement (typically at the center). This may result in a profile intensity line consisting of a plurality of feature values. Additionally, further aggregate features may be extracted from this profile intensity line, and included together with the profile intensity line in the feature map 3006, offering insights into the shape and distribution of the intensity values within the line: examples of such aggregate features may include maximum intensity, depth of the maximum intensity, skewness, kurtosis, standard deviation, and / or entropy. An example is shown in the bottom row of Figure 30. Other features generated based on the ultrasound image may be used alternatively as the US feature map 3006.

[0272] The features 3005 extracted from the DRS spectra and the features 3006 extracted from the US images may be combined as input for the training of a regression model 3009 (or another type of learned model) for tumor margin estimation, as shown in Figure 30. For example, a Random Forest Regression model (as described, for example, in L. Breiman. Random forests, volume 45. Springer, 2001) may be trained. The output 3010 of the model 3009 may be a tumor margin, for example in mm.

[0273] In a study performed by the inventors, to evaluate the added value of this multimodality approach, the prediction performance was assessed using the DRS and US features separately and using the above-described combined model. Before initiating model training, all features were normalized using a min-max scaler, ensuring their values ranged between 0 and 1.

[0274] The performance of the model was evaluated by the mean absolute error (MAE). This metric measured the average vertical distance in millimeters between the true tumor margin and the predicted tumor margin over all measurement locations. A Wilcoxon signed-rank test was used for the statistical analysis of the results, where a p- value <0.05 was considered statistically significant.

[0275] The tumor margin prediction performances for the multi-modality approach and both modalities individually are summarized as follows. Using only DRS data as input, a MAE of 0.84 mm (± 0.16 SD) was achieved for surgical margin estimation, while using only the US data resulted in a MAE of 1.22 mm (± 0.17 SD). In contrast, the combination of features derived from both modalities enhanced the performance, yielding a tumor margin prediction accuracy of 0.76 mm (± 0.13 SD). Herein, SD means standard deviation.

[0276] Fig. 31 shows true tumor margin in millimeters (horizontal axis) versus predicted tumor margin in millimeters (vertical axis). The dots indicate the measurements and the dashed line represents ground truth. A strong and consistent positive correlation was observed between the predicted and true margins (Pearson correlation coefficient = 0.86). Most predictions closely aligned with the true values, especially for the smaller true tumor margins (except for two noticeable outliers). Larger deviations were primarily associated with true tumor margins exceeding 6 mm. Furthermore, no significant difference was found between the predictions of our multi-modality model and the ground truth tumor margins using the Wilcoxon signed-rank test (p-value=0.8547), confirming that the model’s predictions aligned well with the clinical data.

[0277] In certain embodiments, the optical measurements are converted into a feature vector. Certain embodiments involve the ultrasound image in the conversion of the optical measurements into the feature vector. The feature vector may be, for example, a probability value of a particular classification (e.g. a probability that tumor tissue is present at the associated measurement depth) for each optical measurement made, wherein the different optical measurements have different light emitter to light detector distances.

[0278] In the following, an optical contact sensor will be disclosed. This optical contact sensor may be used in the field of non-invasive fiber optic spectroscopy, for example. In particular, in the following, a method and a device to detect optical contact between a fiber optic probe and a sample or subject being investigated are described.

[0279] Optical spectroscopy has been under development as a diagnostic tool in many different fields, such as medicine, veterinary medicine, food industry, forensic sciences and agriculture. Applications developed use a variety of different optical interaction mechanisms to extract information from samples, such as fluorescence, Raman scattering or diffuse reflection. It is not uncommon for these applications to use fiber optics to transport light from a measurement device to a sample or subject and back.

[0280] A common problem occurring when performing such measurements is presented by the quality of the contact between the fiber optic probe and the sample. In fact, the intensity, quality and spectral shape of the measurements are often greatly affected by the distance between fiber optic probe and the sample, the angular orientation of the probe with respect to the sample surface, as well as the refractive index of any material positioned between sample and optical fiber. Such variations may negatively influence optical measurements and may negatively influence the performance of any diagnostic algorithm developed and applied on such measurements. For high-quality, reproducible measurements it is often preferred to perform such measurements in direct contact with the sample. This is especially the case for spatially resolved optical measurements, such as diffuse reflection measurements, but also for single-fiber reflection measurements.

[0281] The problem may be caused at least partially by the fact that, in absence of contact, there is a layer of air between fiber optic probe and sample. The refractive index of air is typically much smaller than the refractive index of the fiber optics and the refractive index of the sample. This may result in a change in the emitting and collecting inclination angle of the light rays with respect to the fiber optics and produces substantial additional reflections at the fiber-to-air and air-to-sample interfaces.

[0282] The present disclosure provides a method and device for rapidly and accurately assessing the presence of proper optical contact between a sample or subject and a fiber optic probe.

[0283] The method and device are is not limited to use during fiber optic measurements, but can be used in any situation where the physical contact between any instrument or tool and any surface is guarded, such as electrical resistance measurements, temperature measurements etc. That is, the optical contact measurement disclosed herein may be used to ensure that two objects are properly contacting each other. After that any measurement may be performed between the two objects, wherein any measurement may be an optical measurement or other any other measurement.

[0284] Fig. 25 depicts a possible embodiment of an optical contact sensor. A light source 2501 produces light, some of which is transported through beam splitter 2502 and optical fiber 2503 to the sample 2504, via the contact surface 2508 at the distal tip of the optical fiber 2503. A signal from the sample (fluorescence, Raman, back scatter etc.) is picked up by fiber 2503 through contact surface 2508, and diverted by the beam splitter 2502 to detector 2505. In addition to the signal from the sample 2504, part of the light signal from the light source 2501 is reflected at the fiber-sample interface at the contact surface 2508. This signal too is diverted by the beam splitter 2502 and detected by the detector 2505.

[0285] Most samples in the fields mentioned above, such as biological samples, will have refractive indices equal or close to that of water (~1 .34), or slightly above. The fiber optic core usually has a slightly higher refractive index (~1.43) than that of water. In case the contact surface 2508 of the optical fiber 2503 is in direct contact with the sample 2504, Fresnel’s equation predicts that the internal reflection coefficient of the optical fiber is approximately equal to 0.001.

[0286] Fig. 26 depicts the same optical contact sensor as Fig. 25, in case of no contact. In this case, a layer of air 2608 is positioned between the contact surface 2508 of the fiber 203 and the surface of the sample 204. As the refractive index of air equals 1 .00, the internal reflection in fiber 203 equals approximately 0.031.

[0287] The light leaving fiber 2503 at the contact surface 2504 will enter the sample 2504 and start interacting with the sample. Some of the light resulting from this interaction will be picked up again by fiber 2503 and reach the detector via the beam splitter 2505. However, as the signal picked up from the sample is usually much smaller than the signal emitted into the sample (by a factor of about 0.001-0.005 for reflectance measurements and much smaller than that for Raman or fluorescence measurements), the signal reaching the detector 2505 in the case of proper optical contact is at least an order of magnitude lower than in the case where there is no proper optical contact. According to the present disclosure, this massive difference in signal intensity may be used to determine the presence of proper optical contact.

[0288] In certain embodiments, the signal intensity measured at the detector 2505 is compared with a threshold, and the result of the comparison is outputted. In certain embodiments, proper contact is detected if the signal intensity is below the threshold, and bad or no contact is detected if the signal intensity is above the threshold. This simple algorithm may be implemented by means of a control unit comprising a processor system (not illustrated). The control unit may be configured to control to perform an action only if the contact is determined to be good. This action may comprise performing a further measurement, such as an optical measurement. Alternatively the control unit may be configured to output an indication of good contact or an indication of no good contact (e.g. an auditive or visual signal perceptible by a user). The threshold may be set as a percentage of the intensity of the light signal generated by the light source 2501. Alternatively, the threshold may be determined empirically by performing a number of measurements with the device with good contact and detecting the signal intensity detected by the detector 2505, and performing a number of measurement with the device with bad or no contact and detecting the signal intensity detected by the detector 2505. The value of the threshold may be set somewhere between the measurements for good contact and the measurements for bad or no contact.

[0289] The embodiments described above can be used as a stand-alone contact sensor, where no additional optical measurements take place.

[0290] Calibration of the device may involve the determination of two reference measurements, one in a situation where optical contact is assured and one where there is no optical contact. These two reference values then allow the definition of an improper-contact-detection-threshold. The detector 2505 can be any detector, such as a photo diode or a spectrometer. A photo diode may be a convenient and cost effective solution.

[0291] In certain embodiments, the setup contains a single fiber measurement system (illumination and detection through the same optical fiber). For example, the detector 2505 may comprise a spectrometer configured to perform the desired measurement. The spectrometer can be used to detect the contact in the manner set forth, but the same spectrometer can also be used to perform spectroscopic analysis of the sample after good contact has been confirmed. Again, to determine the improper-contact- detection-threshold one can take 2 spectra, one with proper contact and one without. One can then define a threshold based on a single wavelength of the spectrum detected by the spectrometer, or based on multiple wavelengths.

[0292] Fig. 27 shows the contact sensor in a multi fiber measurement system. Such a multi fiber measurement system may have illumination and detection through different optical fibers, as elaborated elsewhere in the present disclosure. In use, light source 2501 sends light through a beam splitter 2502 and an optical fiber 2503 to a sample 2504 and illuminates the sample 2504. Some light from the tissue will reach fiber 2503 and be diverted by beam splitter 2502 to the detector 2505. Internal reflection in fiber 2503 will also be diverted by beam splitter 2502 and detected by detector 305. One can, of course, also implement detector 305 as a spectrometer to obtain additional spectral information. Again, to determine the improper-contact-detection-threshold one would need to take 2 spectra, one with proper contact and one without. Moreover, some light from the tissue will reach contact surface 2079 of fiber 2706 and will be detected by spectrometer 2707. When the good contact has been established for fiber 2503, it may be assumed that fiber 2706 is also in good contact with the sample. Alternatively, it is possible to implement a second contact sensor for the second optical fiber 2706, identical to the contact sensor for optical fiber 2503. Then, contact may be established for both optical fibers individually. After that, the contact surface of one of the fibers may be used as the light emitter and the contact surface of the other one of the fibers as the light detector, using methods and configurations as elaborated throughout the present disclosure. Moreover, the concept can be generalized to multi-fiber configurations with more than two fibers, for example the configurations as described herein. The contact sensor may be implemented on one of the fibers, two of the fibers, some of the fibers, or all of the fibers.

[0293] Fig. 28 shows another example of a contact sensor. Technically the contact sensor is identical to the contact sensor of Fig. 27. However, the drawing also shows a housing 2810 of the device and shows that the contact surfaces 2508, 2709 are aligned with the outside surface 2811 of the housing 2810. It will be understood that not all the components need to be present in the same housing. Some components may be present in different housings. In this figure it is shown how the different contact surfaces of the optic fibers are aligned in a plane. However, other alignments are also possible. It will be understood that such an alignment of contact surfaces (with or without a housing) may be applied to and combined with the optical emitter and optical detector as disclosed in embodiments throughout the present disclosure. Moreover, in certain embodiments, the optical fiber(s) may protrude a bit from the outside surface 2811 of the housing 2810 for improved contact. Also, the second optical fiber 2706 and the second optical detector 2707 may be omitted in certain embodiments.

[0294] Certain embodiments comprise a device for determining optical contact, the device comprising a light source, a beam splitter, an optical fiber, and a detector, wherein the device is configured to measure an intensity of the light detected at the detector while emitting light from the light source at the distal tip of the optical fiber. For example, the detected intensity may be compared to a threshold value. The threshold value may be set beforehand based on two measurements performed with and without optical contact. Certain embodiments comprise a method for determining optical contact between a fiberoptic probe and a sample for a fiber optic measurement setup that comprises at least a light source, a beam splitter, an optical fiber, and a detector, using the threshold value. The threshold value may be determined based on two measurements performed with and without optical contact.

[0295] In operation, in the case of good optical contact, as shown in Fig. 25, light from the light source 2501 passes the beam splitter 2502 and enters the optical fiber 2503. At the distal end of the optical fiber there is a small amount of internal reflection that returns to the beam splitter 2502 and is diverted to the detector 2505. Light that is not reflected at the end of the first optical fiber 2503 exits this fiber and enters the sample 2504. A small fraction of the light that entered the sample is picked up again by the optical fiber 2503 and can reach the detector 2505 via the beam splitter 2502.

[0296] In the case of bad optical contact, as shown in Fig. 26, light from the light source 2501 passes the beam splitter 2502 and enters the optical fiber 2503. At the end of the optical fiber, there is a substantial refractive index mismatch between the optic fiber and the environmental air 2608, resulting in a substantial amount of internal reflection that returns to the beam splitter 2502 and is diverted to the detector 2505. Light that is not reflected at the end of the first optical fiber 2503 exits this fiber, passes the air gap 2608 and enters the sample 2504. A small fraction of the light that entered the sample 2504 is picked up by the optical fiber 2503 and can reach the detector 2505 via the beam splitter 2502.

[0297] In the case of the multifiber measurement system, shown in Fig. 27 and Fig. 28, light from the light source 2501 passes the beam splitter 2502 and enters the optical fiber 2503. At the end of the optical fiber 2503, if there is a gap (not illustrated), there is a refractive index mismatch resulting in a high amount of internal reflection that returns to the beam splitter 2502 and is diverted to the detector 2505. If there is no gap, most light is not reflected. Light that is not reflected at the end of the first optical fiber 2503 exits this fiber, passes the possible air gap, and enters the sample 2504. Then the interaction (scattering, Raman scattering, fluorescence etc.) between light and the sample 2504 occurs. The light generated by this interaction spreads through the sample 2504 and a small fraction of this light is picked up by the optical fiber 2503 and can reach the detector 2505 via the beam splitter 2502. In addition, another small fraction is picked up by the second optical fiber 2706 and can reach the second detector 2707. These signals generated in the sample and the information it contains, as measured by the second detector 2707, may have been the reason for performing the measurement. At the same time the first detector 2505 may be used to assure that there is good optical contact between the optical fibers 2503, 2706 and the sample 2504. The amount of internal reflection in fiber 2503 is detected by the first detector 2505 and used to determine the quality of the optical contact between fiberoptic probe (referred to herein as optic scanner) and the sample.

[0298] Fig. 29 illustrates this as a method of contact sensing. The method may optionally begin with bringing 2901 the optical scanner to a measurement location on a sample. Next, the method may proceed by emitting 2902 light through an optical fiber through a contact surface at a tip of the optical fiber of the optical scanner. At the same time, the method may proceed by detecting 2903 an intensity of light received through the contact surface of the tip of the optical fiber. Next, the method may proceed by deciding 2904 whether the contact surface is in good optical contact with the tissue, based on the detected intensity of the received light. Next, or at the same time of step 2903, the method may optionally proceed by performing 2905 an optical measurement using at least two optical fibers: at least one light emitter and at least one light detector. Next, in optional step 2906, the method may proceed by discarding the measurement of step 2905 in case good contact was not detected in step 2904 and / or storing the measurement for further processing in case good contact was detected in step 2904. Advantageously, the tissue has a refractive index that is closer to a refractive index of the optical fiber than to a refractive index of environmental air. This way a clear threshold may be set for the light intensity to be used in step 2904. Steps 2904, 2905 may be performed by a control unit or processor (not illustrated). Moreover, all the steps may be performed under control of the control unit or processor, by exchanging signals where necessary with the light source or light detector.

[0299] Throughout this document, several configurations have been illustrated and explained of a measurement device 901 , the measurement device 901 comprising: an imaging scanner 907, such as an ultrasound scanner, configured to generate an image of a first region of interest 251 ; and an optical scanner 904 comprising at least one light emitter 906 configured to emit light into a second region of interest 252 and at least one light detector 905 configured to detect the light that has interacted with a tissue in the second region of interest 252, to generate optical measurement data; wherein the at least one light emitter 906 and the at least one light detector 905 and the imaging scanner 907 are aligned with each other at least during the measurements, so that the first region of interest 251 overlaps the second region of interest 252. Such a configuration may be highly suitable to perform e.g. a diffuse reflectance spectroscopy analysis in conjunction with an imaging modality such as an ultrasound image generated by an ultrasound scanner. However, it may be important to create a good overlap between the first region of interest and the second region of interest. This overlapping portion of the regions of interest may preferably be close to the scanners, in certain embodiments.

[0300] For example, the imaging scanner and the optical scanner may be configured with parallel axes of operation. The axis of operation being defined as the direction into which a signal is transmitted from the transmitter and the direction from which a signal travels to be received by the detector. Such signal may be e.g. acoustic waves (e.g. for an ultrasound scanner) or optical waves (e.g. for an optical scanner). In general, this direction may be adjusted by orienting the imaging scanner and / or the optical scanner with respect to each other and by manipulating the direction of e.g. optical waves by optic elements. By creating oblique axes of operation, i.e. so that the axes of operation intersect in the overlapping portion of the regions of interest, it is possible to create the overlapping region of interest much closer to the imaging scanner. This can provide important clinical information.

[0301] For example, the imaging scanner 907 may comprise at least one transceiver configured to transmit or receive a signal to and from a first direction to and from the first region of interest, wherein the at least one light emitter 906 and the at least one light detector 905 that detects the diffused light emitted by the at least one light emitter 906 are arranged on one and the same side of the imaging scanner. Herein, “imaging scanner” may denote the transducers of the imaging scanner.

[0302] The at least one light emitter 906 and the at least one light detector 905 may be configured to emit and receive the light to and from a second direction, wherein the second direction is oblique to the first direction. By using obliquely oriented light and acoustic paths, the overlapping portion of the regions of interest may be better controllable. For example, the overlapping portion of the regions of interest may be closer to the imaging scanner. The at least one light emitter 906 and / or the at least one light detector may comprise an optic fiber with a beveled tip. This helps to create an oblique optical path.

[0303] The at least one light emitter 906 and / or the at least one light detector 905 may comprise an optic fiber, wherein the tip of the optic fiber is arranged in a direction that is oblique to the first direction. Such an oblique tip helps to create an oblique optical path.

[0304] The at least one light emitter 906 and the at least one light detector 905 may extend further in the first direction than the transducers of the imaging scanner 907, wherein a gap in the first direction between the transducers of the imaging scanner 907 and the extension of the at least one light emitter 906 and the at least one light detector 905 is filled with a couplant medium for the signals transmitted and received by the transducers of the imaging scanner. This also improves the measurement of superficial structures by the imaging scanner in conjunction with the optical scanner.

[0305] The medical system may comprise an actuator configured to move the imaging scanner 907 and the optical scanner 904 over a surface to create the overlap between the first region of interest 251 and the second region of interest 252 which are scanned sequentially by the imaging scanner 907 and the optical scanner 904, respectively. This feature provides an alternative way to obtain overlapping regions of interest, by moving the medical system in between the imaging scanner operation and the optical scanner operation, so that they can successively operate on the same region of interest.

[0306] The medical system may comprise a motion detector configured to detect a movement of the medical system in between a measurement of the first region of interest

[0307] 251 by the imaging scanner 907 and a measurement of the second region of interest

[0308] 252 by the optical scanner 904. The motion detector may output an indication of a movement (e.g. a movement vector) to the control unit. The control unit can calculate whether there is an overlap between the first region of interest sampled by the image scanner and the second region of interest sampled by the optical scanner, using the motion information.

[0309] Figs. 32-39 show several configurations of optical fibers in relation to an ultrasound scanner. Figs. 32-37 show side views. Figs. 38 and 39 show perspective views. These optical fibers could be comprised in the optical scanner 1403. As shown in the diagrams of Fig. 9 and Fig. 14, the arrangements of emitters / detectors shown in the other figures (e.g. Fig. 1 , 2, 10, 11 , 12, 13,15), could all be tips of optical fibers. Although in Figs. 32-39 the imaging scanner is an ultrasound scanner, this is not a limitation. The ultrasound scanner may be replaced by another imaging scanner as described elsewhere in the present disclosure.

[0310] Fig. 32 shows an example of a distal end of an optical fiber 3201. In certain embodiments, the tips of the optical fibers of the optical scanner 904 can be similar to the tip 3205 of optical fiber 3201 . Tip 3205 of optical fiber 3201 has a beveled surface 3202. This beveled surface 3202 causes the light beam emitted by the optical fiber 3201 to de deflected to a direction 3203 with an oblique angle to the central axis 3204 of the optical fiber 3201 .

[0311] Fig. 33 shows the optical fiber 3201 together with an ultrasound scanner 3301. The ultrasound scanner has an acoustic sampling volume 3304 along an acoustic path 3303 that extends straight from the ultrasound scanner 3301. The optical scanner has optical fiber 3201 arranged parallel to the ultrasound scanner 3301. However, the light path 3203 of the optical fiber 3201 is not a straight extension of the central axis of the optical fiber 3201 , but rather has an oblique direction compared to the acoustic path 3303 of the ultrasound scanner 3301. Therefore, also the optical sampling volume 3206 is oriented oblique to the acoustic sampling volume 3304. This oblique direction influences where the overlapping region 3305 where the optical sampling volume 3206 and the acoustic sampling volume 3304 overlap is located. Herein, ‘sampling volume’ and ‘region of interest’ have the same meaning and these terms may be used interchangeably.

[0312] It will be understood that in Fig. 32-33, the arrows indicate the direction of emitted optical beams. The arrows may be inverted to indicate the direction of incoming light signals collected by the optical detector.

[0313] Since light is deflected by the sampling volume back towards the optical scanner, the optical detector and the optical emitter can both be located on the same side of the ultrasound scanner. This can be done regardless of whether the tip 3205 of the optical fibers 3201 are beveled. Preferably, the beveled surface 3202 of the tip of the fiber 3201 faces obliquely away from the ultrasound scanner 3301 . The arrangement of optical detectors on the same side of the acoustic scanner allows optical detectors and optical emitters to be arranged closely together, so that a second region of interest can be realized close to the optical scanner. This also can help to bring the overlapping part of the regions of interest closer to the ultrasound scanner 3301 . Figs. 34 to 36 show examples in which the ultrasound scanner is oblique to the optical scanner.

[0314] In Fig. 34, the ultrasound scanner is oblique to the surface 3401 of the sample (e.g. the skin surface). That is, the transducers of the ultrasound scanner 3301 are positioned to transmit and receive their signals under an oblique angle. A gap between the transducers of the ultrasound scanner 3301 and the surface 3401 may be filled with a contact gel. Alternatively, the medical system may comprise acoustically conducting filling material 3402. The optical fiber 3201 is in this case perpendicular to the sample surface 3401. The optical fiber 3201 optionally has a beveled tip 3202. Any space between the tip 3202 of the optical fiber 3201 and the surface 3401 of the sample may be filled with optically transparent contact gel and / or by an optical conductor (taking into account any deflection index). The oblique orientation of the transducers of the ultrasound scanner 3301 causes the acoustic sampling volume 3304 to be obliquely oriented towards the optical sampling volume 3206. If the optical fiber 3201 has a beveled surface 3202, the optical sampling volume may be obliquely oriented towards the acoustical sampling volume 3304.

[0315] Fig. 35 shows a configuration in which the ultrasound scanner is straight with respect the skin surface 3401 , that is, the ultrasound scanner is configured to emit acoustic signals orthogonally into the sample. The optical fiber 3201 is oriented oblique to the ultrasound scanner 3301 , so that the optical sampling volume 3206 tends towards the acoustic sampling volume 3304. If the optical fiber 3201 has a beveled tip surface 3202 (as illustrated), the optical sampling volume 3206 tends even more towards the acoustic sampling volume.

[0316] Fig. 36 shows a configuration in which both the ultrasound scanner 3301 and the optical fiber 3201 are obliquely oriented towards the sample surface 3401. Any space between the scanners and the sample surface 3401 can be filled with gel or filler material 3402, as described above. In this configuration also the beveled surface 3202 of the optical fiber 3201 is optional. The optical sampling volume 3206 and the acoustic sampling volume 3304 tend towards each other.

[0317] In all of Figs. 34 to 36, the acoustic scanner (i.e. the acoustic transducers) are arranged oblique to the optical fibers 3201 of the optical scanner.

[0318] Fig. 37 shows an alternative example, in which the ultrasound scanner 3301 and the optical scanner 3201 are arranged parallel to each other, so that the acoustic sampling volume 3304 and the optical sampling volume 3206 exist parallel to each other, wherein the ultrasound transducers and the optical fibers are a distance d apart from each other. While scanning, the system comprising the ultrasound scanner 3301 and the optical fibers 3201 is moved with a speed v having a direction as shown by arrow v in Fig. 37. By combining the ultrasound measurement performed at a time t with the optical measurement made at a time t+d / v, optical and acoustic measurements are obtained with overlapping acoustic sampling volume 3304 and optical sampling volume 3206.

[0319] For example, the distance traveled by the device over the sampling surface 3401 can be measured by a motion detector 3702, such as an accelerometer or an additional optical movement detector or a trackball (similar to a motion detector in a computer mouse pointer). A computer system (e.g. control unit) can receive the measured ultrasound and optical data and the measured movement of the device in between these measurements, and can associate acoustical measurements with the optical measurements made from the same location and / or with overlapping regions of interest.

[0320] Fig. 38 shows a perspective view of an example medical device as shown in Figs. 32-36. The figure shows that the optical fibers 3201 can be arranged on one side of the ultrasound scanner 3301 , preferably in a row alongside the ultrasound scanner 3301. The optical fibers 3201 can be arranged with alternating emitter / detector pattern as in Fig. 1 E, or in a pattern with a single detector that cooperates with several emitters, as in Fig. 1C. Other patterns are equally possible. For example, the emitters and detectors could be arranged in two adjacent rows rather than one row.

[0321] Fig. 39 shows that the fiber tips 3205 can be displaced towards the envisaged sampling surface 3401 (i.e., in the scanning direction), by a height h. Thus, the ultrasound scanner 3301 is further away from the envisaged sampling surface 3401. This space in between the ultrasound scanner 3301 and the surface 3401 of the sample can be filled up by couplant gel or a (solid) couplant filler material 3402.

[0322] Fig. 40 shows a configuration in which the optical fibers are oriented obliquely, similar to Fig. 35. However, in Fig. 40 the ultrasound scanner 3301 is further away, by a height h, from the envisaged sampling surface 3401. The optical fibers 3201 are oriented oblique and extend in the space in between the ultrasound scanner 3301 and the envisaged sampling surface 3401. Thus, the tips 3205 of the optical fibers 3201 may be very close to or even reach into the acoustic sampling volume. This allows to highly specifically placing the optical sampling volume with respect to the acoustic sampling volume. This may be particularly advantageous in combination with the alternating emitter and detector pattern of the fibers.

[0323] In the above description, the direction in which signals are emitted, may be the dominant direction of an emitted beam. There may always leak signals towards a broader region. The direction in which signals are said to be received, may be a direction to which transducers are most sensitive by virtue of their design and orientation. The direction in which optical signals are received, may be a direction to which the optical detector is most receptive / sensitive, by virtue of design (e.g. tip design, possible additional optical elements such as lenses) and orientation of the tip of optical detector or the optical fiber. However, it will be understood that acoustic or optical detectors may be agnostic to the direction of incoming signals. They may be able to detect any signal that arrives at the detector. In that case, the direction of emission of acoustic / optical signals is what determines the sampling volume.

[0324] Some or all aspects of the invention may be suitable for being implemented in form of software, in particular a computer program product. The computer program product may comprise a computer program stored on a non-transitory computer- readable media. Also, the computer program may be represented by a signal, such as an optic signal or an electro-magnetic signal, carried by a transmission medium such as an optic fiber cable or the air. The computer program may partly or entirely have the form of source code, object code, or pseudo code, suitable for being executed by a computer system. For example, the code may be executable by one or more processors.

[0325] The examples and embodiments described herein serve to illustrate rather than limit the invention. The person skilled in the art will be able to design alternative embodiments without departing from the spirit and scope of the present disclosure, as defined by the appended claims and their equivalents. Reference signs placed in parentheses in the claims shall not be interpreted to limit the scope of the claims. Items described as separate entities in the claims or the description may be implemented as a single hardware or software item combining the features of the items described.

[0326] Certain aspects are defined in the following clauses.

[0327] 1. A medical system comprising a measurement device (901), the measurement device (901) comprising: an imaging scanner (907), such as an ultrasound scanner, configured to generate an image of a first region of interest (251); an optical scanner (904) comprising at least one light emitter (906) configured to emit light into a second region of interest (252) and at least one light detector (906) configured to detect the light that has interacted with a tissue in the second region of interest (252), to generate optical measurement data, wherein the at least one light emitter (906) and the at least one light detector (905) and the imaging scanner (907) are aligned with each other at least during the measurements, so that the first region of interest (251) overlaps the second region of interest (252).

[0328] 2. The medical system of any preceding clause, wherein the imaging scanner (907) and the optical scanner (904) are configured to operate simultaneously.

[0329] 3. The medical system of any preceding clause, further comprising a display (1101) configured to output the image and an indication (1121) of the second region of interest (252) in the image.

[0330] 4. The medical system of any preceding clause, comprising a control unit (902) configured to control to perform a diffuse reflectance spectroscopy analysis on the optical measurement data and / or wherein the imaging scanner (907) comprises an ultrasound scanner and the image generated by the imaging scanner comprises an ultrasound image.

[0331] 5. The medical system of any preceding clause, wherein the at least one light emitter (101-108) and the at least one light detector (110) form a plurality of light emitter to light detector distances.

[0332] 6. The medical system of clause 5, wherein the medical system (901) is configured to sequentially or simultaneously activate different pairs of one light emitter of the at least one light emitter (611) and one light detector of the at least one light detector (612), wherein the different pairs are associated with different light emitter to light detector distances.

[0333] 7. The medical system of clause 5 or 6, wherein the optical scanner (904) comprises at least a light emitter to light detector distance of at most 2 millimeters; a light emitter to light detector distance of at least 6 millimeters; and a light emitter to light detector distance that is larger than 2 millimeters and smaller than

[0334] 6 millimeters.

[0335] 8. The medical system of any preceding clause, wherein the optical scanner (904) comprises at least 6 emitter to detector distances

[0336] 9. The medical system of any preceding clause, wherein the optical scanner (904) is aligned in a center of the imaging scanner (907), so that the second region of interest (252) is registered to a middle of the first region of interest (251).

[0337] 10. The medical system of any preceding clause, wherein the at least one light emitter and the at least one light detector are arranged interleaved, in a row, to form a plurality of interleaved light emitters and light detectors.

[0338] 11 . The medical system of any preceding clause, further comprising a control unit (902) configured to estimate a measurement depth of the optical scanner (904) based on the image.

[0339] 12. The medical system of any preceding clause, further comprising a control unit (902) configured to detect a transition from a first tissue layer with a first tissue type to a second tissue layer with a second tissue type, based on the image and / or the optical measurement data.

[0340] 13. The medical system of any preceding clause, further comprising a learned model configured to combine the image and the optical measurement data, which are registered to each other, to estimate an estimated tissue type classification and / or a depth estimation of a tissue type.

[0341] 14. The medical system of clause 13, wherein the learned model is configured to estimate, for a particular measurement comprising an image generated by the imaging scanner (907) and an optical measurement by the optical scanner (904), a measurement depth associated with at least one light emitter to light detector distance of the optical scanner (904), based on the image.

[0342] 15. The medical system of any preceding clause, further comprising a clip- on unit (222) configured to clip the optical scanner (904), detachably and at a reproduceable relative position, to the imaging scanner (907).

[0343] 16. A medical system configured to calculate a distance from an image scanner (907) to a tissue layer, the system comprising: a learned model (1500) configured to process image data generated by an imaging scanner (907), the learned model (1500) comprising a backbone (1501) connected to a head architecture (1510); and a processor system (902) configured to estimate a distance from the imaging scanner (907) to a tissue layer by applying the learned model (1500) to at least one image generated by the imaging scanner (907), wherein the head architecture (1510) is configured to output a direct estimation of the distance to the tissue layer, based on a feature map generated by the backbone (1501).

[0344] 17. The medical system of clause 16, wherein the head architecture (1510) comprises: a horizontal mean pooling layer (1506) configured to pool the feature map, and a one-dimensional convolution (1507) operating on the horizontal mean pooling layer.

[0345] 18. A combined measurement method, comprising: providing (2201) an imaging scanner aligned with an optical scanner, the optical scanner comprising a light emitter and a light detector; generating (2202) an image of a first region of interest (251) using the imaging scanner; emitting (2203) light into a second region of interest (252) using the light emitter and detecting (2204) the light that has interacted with a tissue in the second region of interest (252) using the light detector, wherein the imaging scanner is aligned with the optical scanner so that the first region of interest (251) overlaps the second region of interest (252), and the first region of interest (251) is registered with the second region of interest (252).

[0346] 19. A computer-implemented measurement method, comprising: receiving (2301) an image of a first region of interest (251) from an imaging scanner; receiving (2302) light measurement data corresponding to an optical measurement of a second region of interest (252), wherein the first region of interest (251) overlaps the second region of interest (252); accessing (2303) parameters of a registration of the first region of interest (251) with the second region of interest (252); and detecting (2304) a tissue type in the second region of interest (252) based on the light measurement data and estimating (2305) a distance to the tissue type based on the image, and relating (2306) the detected tissue type to the estimated distance of the tissue type based on the parameters of the registration.

[0347] 20. A computer-implemented image analysis method, comprising: receiving (2401) an image of a first region of interest (251), wherein the image is generated by an imaging scanner; and estimating (2402) a tumor margin using a learned model configured to process the image generated by the imaging scanner, the learned model comprising a backbone connected to a head architecture, wherein the head architecture outputs a direct estimation of the distance to the tissue layer, based on a feature map generated by the backbone.

[0348] 21. A method of training a model to classify optical measurement data, the method comprising: providing a training set (400) comprising pairs of measurements, each pair comprising an image of a first region of interest (251) of a sample and an optical measurement of a second region of interest (252) of the sample, wherein the first region of interest (251) is registered to the second region of interest (252) and the first region of interest (251) overlaps the second region of interest (252); determining (402) at least one parameter representing an uncertainty regarding measurements stored in the training set; selecting (403) a plurality of combinations of values of the at least one parameter; assigning (404) a classification label to the optical measurement of each pair for each selected combination of values of the at least one parameter, wherein the classification label is extracted from the image of each pair under the assumption of the respective selected combination of values of the at least one parameter; and training (405) a model for each selected combination of values of the at least one parameter, using the classification labels corresponding to the respective selected combination of values of the at least one parameter as target output values for the model.

[0349] 22. The method of clause 21 , wherein the parameter relates to a measurement depth corresponding to the second region of interest (252) of the sample. 23. The method of clause 21 or 22, wherein the step of assigning (404) the classification label comprises: estimating a distance of a tissue type based on the image; setting an assumed sampling depth of the optical measurement; and comparing the depth of the tissue type in the image to the assumed sampling depth.

[0350] 24. The method of any one of clauses 21 to 23, wherein the at least one parameter comprises at least one of: a depth bias (301) that augments an estimated measurement depth of the first measurement modality, a depth uncertainty margin (302) used to exclude measurements where a difference between the estimated distance to the tissue type in the image and the estimated sampling depth is within the depth uncertainty margin; a width (303) of a subregion of the first region of interest (251) that is used to calculate an average distance to the tissue type in the image; and a scaling factor (304) that scales the width and the depth uncertainty margin.

[0351] 25. A contact sensor comprising: a light source (2501); a light detector (2505); an optical fiber (2503), wherein a proximal end of the optical fiber (2503) is optically coupled to the light source (2501) and to the optical detector (2505) and the distal tip of the optical fiber (2503) comprises a contact surface (2508); and a control unit configured to detect whether the contact surface (2508) is in contact with a tissue, based on a signal intensity of a light signal detected at the optical detector (2505) while the light source emits light into the optical fiber (2503).

[0352] 26. The contact sensor of clause 25, further comprising a beam splitter at a proximal tip of the optical fiber (2503), wherein the beam splitter is configured to pass the light from the light source (2501) into the optical fiber (2503) and to diverge light traveling in the direction from the distal tip of the optical fiber (2503) to the proximal tip of the optical fiber (2503) to the light detector (2505).

[0353] 27. The contact sensor of clause 25 or 26, further comprising a second light detector (2707); and a second optic fiber (2706), wherein a proximal end of the second optical fiber (2706) is optically connected to the second light detector, and the distal tip of the second optical fiber (2706) comprises a second contact surface (2709); wherein the contact surface (2508) of the optical fiber (2503) and the second contact surface (2709) of the second optical fiber (2706) are aligned in a plane.

[0354] 28. The contact sensor of any one of clauses 25 to 27, wherein at least a distal portion of the optical fiber (2503) is included in a housing (2810) having an outside surface (2811), and the contact surface (2508) is aligned with the outside surface (2811) of the housing (2810), and optionally the second contact surface (2709) is also aligned with the outside surface (2811) of the housing (2810).

[0355] 29. A method of contact sensing, comprising emitting (2902) light through an optical fiber through a contact surface at a tip of the optical fiber; detecting (2903) an intensity of light received through the contact surface of the tip of the optical fiber; and determining (2904) whether the contact surface is in contact with a tissue, based on the detected intensity of the received light.

[0356] 30. The method of clause 29, wherein the tissue has a refractive index that is closer to a refractive index of the optical fiber than to a refractive index of environmental air.

Claims

CLAIMS:

1. A medical system comprising a measurement device (901), the measurement device (901) comprising: an imaging scanner (907), such as an ultrasound scanner, configured to generate an image of a first region of interest (251); an optical scanner (904) comprising at least one light emitter (906) configured to emit light into a second region of interest (252) and at least one light detector (905) configured to detect the light that has interacted with a tissue in the second region of interest (252), to generate optical measurement data; and a control unit (902) configured to control to perform a diffuse reflectance spectroscopy analysis on the optical measurement data, wherein the at least one light emitter (906) and the at least one light detector (905) and the imaging scanner (907) are aligned with each other at least during the measurements, so that the first region of interest (251) overlaps the second region of interest (252).

2. The medical system of claim 1 , wherein the at least one light emitter (906) and the at least one light detector (905) are arranged on one side of the imaging scanner.

3. The medical system of claim 2, wherein the imaging scanner (907) comprises at least one transceiver configured to transmit a signal in a first direction to the first region of interest; wherein the at least one light emitter (906) is configured to emit the light in a second direction, to the second region of interest, wherein the second direction is oblique to the first direction.4 The medical system of claim 3, wherein the at least one light emitter (906) and / or the at least one light detector comprise an optic fiber with a beveled tip.

5. The medical system of claim 3 or 4, wherein the at least one light emitter (906) and / or the at least one light detector (905) comprise an optic fiber, wherein the tip of the optic fiber is arranged in a direction that is oblique to the first direction.

6. The medical system of any preceding claim, wherein the at least one light emitter (906) and the at least one light detector (905) extend further in the first direction than transducers of the imaging scanner (907).

7. The medical system of claim 2, wherein the medical system comprises an actuator configured to move the imaging scanner (907) and the optical scanner (904) in between a measurement of the first region of interest by the imaging scanner and a measurement of the second region of interest by the optical scanner to create the overlap between the first region of interest (251) and the second region of interest (252), or wherein the medical system comprises a movement detector configured to detect a movement of the medical system in between a measurement of the first region of interest(251) by the imaging scanner (907) and a measurement of the second region of interest(252) by the optical scanner (904).

8. The medical system of any preceding claim, wherein the imaging scanner (907) and the optical scanner (904) are configured to operate simultaneously.

9. The medical system of any preceding claim, further comprising a display (1101) configured to output the image and an indication (1121) of the second region of interest (252) in the image.

10. The medical system of any preceding claim, wherein the imaging scanner (907) comprises an ultrasound scanner and the image generated by the imaging scanner comprises an ultrasound image.11 . The medical system of any preceding claim, wherein the at least one light emitter (101-108) and the at least one light detector (110) form a plurality of light emitter to light detector distances.

12. The medical system of claim 5, wherein the at least one light emitter (906) comprises a plurality of light emitters, wherein each of the plurality of light emitters is configured to form a pair with a same light detector of the at least one light detector (905), or wherein the at least one light detector (905) comprises a plurality of light detectors, wherein each of the plurality of light detectors is configured to form a pair with a same light emitter of the at least one light emitter (906).

13. The medical system of claim 12, wherein the at least one light emitter (906) and the at least one light detector (905) are arranged in a line.

14. The medical system of any one of claims 5 to 7, wherein the medical system (901) is configured to sequentially or simultaneously activate different pairs of one light emitter of the at least one light emitter (611) and one light detector of the at least one light detector (612), wherein the different pairs are associated with different light emitter to light detector distances.

15. The medical system of any one of claims 5 to 8, wherein the optical scanner (904) comprises at least a light emitter to light detector distance of at most 2 millimeters; a light emitter to light detector distance of at least 6 millimeters; and a light emitter to light detector distance that is larger than 2 millimeters and smaller than 6 millimeters.

16. The medical system of any preceding claim, wherein the optical scanner (904) comprises at least 6 emitter to detector distances.

17. The medical system of any preceding claim, wherein the optical scanner (904) is aligned in a center of the imaging scanner (907), so that the second region of interest (252) is registered to a middle of the first region of interest (251).

18. The medical system of any preceding claim, wherein the at least one light emitter and the at least one light detector are arranged interleaved, in a row, to form a plurality of interleaved light emitters and light detectors.

19. The medical system of any preceding claim, further comprising a control unit (902) configured to estimate a measurement depth of the optical scanner (904) based on the image.

20. The medical system of any preceding claim, further comprising a control unit (902) configured to detect a transition from a first tissue layer with a first tissue type to a second tissue layer with a second tissue type, based on the image and / or the optical measurement data.21 . The medical system of any preceding claim, further comprising a learned model configured to combine the image and the optical measurement data, which are registered to each other, to estimate an estimated tissue type classification and / or a depth estimation of a tissue type.

22. The medical system of claim 21 , wherein the learned model is configured to estimate, for a particular measurement comprising an image generated by the imaging scanner (907) and an optical measurement by the optical scanner (904), a measurement depth associated with at least one light emitter to light detector distance of the optical scanner (904), based on the image.

23. The medical system of any preceding claim, further comprising a clip-on unit (222) configured to clip the optical scanner (904), detachably and at a reproduceable relative position, to the imaging scanner (907).

24. A medical system configured to calculate a distance from an image scanner (907) to a tissue layer, the system comprising: a learned model (1500) configured to process image data generated by an imaging scanner (907), the learned model (1500) comprising a backbone (1501) connected to a head architecture (1510); anda processor system (902) configured to estimate a distance from the imaging scanner (907) to a tissue layer by applying the learned model (1500) to at least one image generated by the imaging scanner (907), wherein the head architecture (1510) is configured to output a direct estimation of the distance to the tissue layer, based on a feature map generated by the backbone (1501).

25. The medical system of claim 24, wherein the head architecture (1510) comprises: a horizontal mean pooling layer (1506) configured to pool the feature map, and a one-dimensional convolution (1507) operating on the horizontal mean pooling layer.

26. A combined measurement method, comprising: providing (2201) an imaging scanner aligned with an optical scanner, the optical scanner comprising a light emitter and a light detector; generating (2202) an image of a first region of interest (251) using the imaging scanner; emitting (2203) light into a second region of interest (252) using the light emitter and detecting (2204) the light that has interacted with a tissue in the second region of interest (252) using the light detector; and performing a diffuse reflectance spectroscopy analysis on the optical measurement data, wherein the imaging scanner is aligned with the optical scanner so that the first region of interest (251) overlaps the second region of interest (252), and the first region of interest (251) is registered with the second region of interest (252).

27. A computer-implemented measurement method, comprising: receiving (2301) an image of a first region of interest (251) from an imaging scanner; receiving (2302) light measurement data corresponding to an optical measurement of a second region of interest (252), wherein the first region of interest (251) overlaps the second region of interest (252);accessing (2303) parameters of a registration of the first region of interest (251) with the second region of interest (252); and detecting (2304) a tissue type in the second region of interest (252) based on the light measurement data and estimating (2305) a distance to the tissue type based on the image, and relating (2306) the detected tissue type to the estimated distance of the tissue type based on the parameters of the registration.

28. A computer-implemented image analysis method, comprising: receiving (2401) an image of a first region of interest (251), wherein the image is generated by an imaging scanner; and estimating (2402) a tumor margin using a learned model configured to process the image generated by the imaging scanner, the learned model comprising a backbone connected to a head architecture, wherein the head architecture outputs a direct estimation of the distance to the tissue layer, based on a feature map generated by the backbone.

29. A method of training a model to classify optical measurement data, the method comprising: providing a training set (400) comprising pairs of measurements, each pair comprising an image of a first region of interest (251) of a sample and an optical measurement of a second region of interest (252) of the sample, wherein the first region of interest (251) is registered to the second region of interest (252) and the first region of interest (251) overlaps the second region of interest (252); determining (402) at least one parameter representing an uncertainty regarding measurements stored in the training set; selecting (403) a plurality of combinations of values of the at least one parameter; assigning (404) a classification label to the optical measurement of each pair for each selected combination of values of the at least one parameter, wherein the classification label is extracted from the image of each pair under the assumption of the respective selected combination of values of the at least one parameter; and training (405) a model for each selected combination of values of the at least one parameter, using the classification labels corresponding to the respective selectedcombination of values of the at least one parameter as target output values for the model.

30. The method of claim 29, wherein the parameter relates to a measurement depth corresponding to the second region of interest (252) of the sample.

31. The method of claim 29 or 30, wherein the step of assigning (404) the classification label comprises: estimating a distance of a tissue type based on the image; setting an assumed sampling depth of the optical measurement; and comparing the depth of the tissue type in the image to the assumed sampling depth.

32. The method of any one of claims 29 to 31 , wherein the at least one parameter comprises at least one of: a depth bias (301) that augments an estimated measurement depth of the first measurement modality, a depth uncertainty margin (302) used to exclude measurements where a difference between the estimated distance to the tissue type in the image and the estimated sampling depth is within the depth uncertainty margin; a width (303) of a subregion of the first region of interest (251) that is used to calculate an average distance to the tissue type in the image; and a scaling factor (304) that scales the width and the depth uncertainty margin.

33. A contact sensor comprising: a light source (2501); a light detector (2505); an optical fiber (2503), wherein a proximal end of the optical fiber (2503) is optically coupled to the light source (2501) and to the optical detector (2505) and the distal tip of the optical fiber (2503) comprises a contact surface (2508); and a control unit configured to detect whether the contact surface (2508) is in contact with a tissue, based on a signal intensity of a light signal detected at the optical detector (2505) while the light source emits light into the optical fiber (2503).

34. The contact sensor of claim 33, further comprising a beam splitter at a proximal tip of the optical fiber (2503), wherein the beam splitter is configured to pass the light from the light source (2501) into the optical fiber (2503) and to diverge light traveling in the direction from the distal tip of the optical fiber (2503) to the proximal tip of the optical fiber (2503) to the light detector (2505).

35. The contact sensor of claim 33 or 34, further comprising a second light detector (2707); and a second optic fiber (2706), wherein a proximal end of the second optical fiber (2706) is optically connected to the second light detector, and the distal tip of the second optical fiber (2706) comprises a second contact surface (2709); wherein the contact surface (2508) of the optical fiber (2503) and the second contact surface (2709) of the second optical fiber (2706) are aligned in a plane.

36. The contact sensor of any one of claims 33 to 35, wherein at least a distal portion of the optical fiber (2503) is included in a housing (2810) having an outside surface (2811), and the contact surface (2508) is aligned with the outside surface (2811) of the housing (2810), and optionally the second contact surface (2709) is also aligned with the outside surface (2811) of the housing (2810).

37. A method of contact sensing, comprising emitting (2902) light through an optical fiber through a contact surface at a tip of the optical fiber; detecting (2903) an intensity of light received through the contact surface of the tip of the optical fiber; and determining (2904) whether the contact surface is in contact with a tissue, based on the detected intensity of the received light.

38. The method of claim 37, wherein the tissue has a refractive index that is closer to a refractive index of the optical fiber than to a refractive index of environmental air.

Images