Apparatus, method and computer program

Abstract

Examples relate to a device that includes one or more processors and one or more memory devices. The device is configured to receive reference pattern data indicative of a first image of a pattern from a first channel, and to receive threshold data indicative of a plurality of threshold values ​​for performing a distance transformation. The device is further configured to generate reference data using at least two threshold values ​​from the plurality of threshold values ​​by applying the distance transformation to the reference sample data. The device is further configured to receive sample data indicative of a second image of the sample from a second channel different from the first channel, and to generate structural image data indicative of a structure of the sample. The structural image data is generated based on the reference data and the sample data.

Description

Technical area

[0001] Examples include a device, a method, and a computer program. background

[0002] In certain spatial analysis contexts, it is important to modulate biomarker expression signals by either enhancing or suppressing them depending on the proximity or distance between objects labeled with different biomarkers. For example, examining the spatial relationships between individual cells, as indicated by biomarker expression in multiplexed images, can provide valuable insights into potential cell interactions, associations, and spatial arrangements. Although segmentation techniques are commonly used for this purpose, they can encounter challenges, particularly with multiplexed biomarker image sets characterized by complicated structures, noisy backgrounds, or quality issues. Therefore, there may be a desire for an improved approach to analyzing a multiplexed biomarker image set.

[0003] US 2018 / 0 240 239 A1 discloses a method for improving segmentation and classification in multi-channel images. A distance transformation image is calculated from the binary mask to generate a distance transformation image.

[0004] ZHANG, Chao, [et al.]: Clustered nuclei splitting via curvature information and gray-scale distance transform discloses an algorithm based on curvature information, gray-scale distance transform, and shortest path, which can make full use of the information on concavity and image intensity to find markers, each of which represents an individual object, and accurate dividing lines between objects using the shortest path and intersection matching. Summary

[0005] This wish is met by the subject matter of the independent claims.

[0006] The concept proposed in the present disclosure is based on the recognition that structural image data indicative of a sample structure can be generated using reference data and sample data. The reference data can be generated by applying distance transformation using multiple thresholds. Based on the distance transformation, a distance-transformed reference image can be created to enhance or suppress pixel values ​​of a target image of a sample. In this way, a segmentation-free process can be provided to enhance or suppress pixel values ​​in a target image of a sample. In this way, the visibility of a sample structure can be improved.

[0007] Examples provide an apparatus comprising one or more processors and one or more memory devices. The apparatus is configured to obtain reference pattern data indicative of a first image of a pattern from a first channel, and to obtain threshold data indicative of a plurality of threshold values ​​for performing a distance transformation. The apparatus is further configured to generate reference data using at least two threshold values ​​from the plurality of threshold values ​​by applying the distance transformation to the reference sample data. The apparatus is further configured to obtain sample data indicative of a second image of the sample from a second channel different from the first channel, and to generate structural image data indicative of a structure of the sample. The structural image data is generated based on the reference data and the sample data.Applying the distance transformation using at least two thresholds can improve the accuracy and robustness of the distance transformation. Thus, the distance transformation can be used in conjunction with the sample data to generate structural image data, even for complex structures. Furthermore, the at least two thresholds can increase the usability of the reference sample data and thus improve the reference data. In this way, the susceptibility to errors during the generation of structural image data can be reduced.

[0008] In one example, generating the reference data may involve applying a mapping function to a distance-transformed first image of the sample. The mapping function may enable pixel values ​​in the second image to be enhanced or suppressed based on the distance-transformed first image. For example, various functions such as monotonically decreasing functions or monotonically increasing functions may be used as the mapping function. In this way, the pixel values ​​in the second image can be adjusted as desired.

[0009] In one example, the mapping function can be a monotonically decreasing function, a monotonically increasing function, and / or a bandpass function. In this way, the representation or visibility of the sample's structure can be adjusted as desired.

[0010] In one example, the distance transformation can be a grayscale distance transformation. A grayscale distance transformation can enable simpler and more efficient computation than a standard distance transformation, which requires a binary segmentation mask as input. This allows the structural image data to be generated more easily.

[0011] In one example, the device may be further configured to generate overlay image data indicating the structure of the image. The overlay image data is generated based on the structure image data and the reference data. The overlay image data may enable enhanced representation or visibility of the sample's structure.

[0012] In one example, one of the plurality of thresholds for performing the distance transformation may be an intensity threshold for retaining or excluding pixels for performing the distance transformation. That is, the threshold may define the pixel value above which a pixel in the first image may be considered background or foreground.

[0013] In one example, the device may be configured to generate intermediate binary data indicative of a plurality of binary-like representations or a plurality of binary masks for performing the distance transformation. The intermediate binary data is generated based on the reference pattern data and the threshold data. Further, generating the reference data may include applying the distance transformation to the binary data to generate a plurality of distance-transformed first images. Using a plurality of binary-like representations or a plurality of binary masks may enable the generation of average distance-transformed data (indicative of an average distance-transformed first image) by generating a plurality of intermediate distance-transformed first images.The averaged distance-transformed data can improve the efficiency, accuracy, robustness, and / or interpretability of spatial analysis tasks. In this way, structural image data can be improved.

[0014] In one example, the first channel and the second channel may be part of a multiplexed image set. This means that the reference sample data and the sample data can be obtained simultaneously. Therefore, the device can generate the structural image data based on a multiplexed image set of a sample.

[0015] In one example, the generation of structural image data may be a segmentation-free process. This may simplify and / or improve the generation of structural image data.

[0016] In one example, the generation of the structural image data may retain an unsegmented feature of the first image of the sample and the second image of the sample. This means that the structural image data may be generated without segmentation. This avoids error-prone segmentation.

[0017] In one example, generating the structural image data may include highlighting a portion of the second image corresponding to a portion of the reference data with a small distance and / or attenuating a portion of the second image corresponding to a portion of the reference data with a large distance. Thus, a distance transformation may be used to generate structural image data. The use of the distance transformation may make it possible to avoid segmentation of the first image and / or the second image.

[0018] In one example, generating the structural image data may include highlighting a spatial feature in the example image data that corresponds to a structure represented by distance values ​​in the reference data. For example, the spatial feature, e.g., a structure, may be highlighted based on the representation of the structure in the first image. This means that no segmentation is required to highlight the spatial feature in the second image.

[0019] In one example, the reference pattern data may be raw reference pattern data from the first image of the sample, and the distance transformation to generate the reference data is applied to the raw reference pattern data. This means that the distance transformation can be applied directly to the reference sample data without further preprocessing. This can simplify the generation of the reference data.

[0020] In one example, the sample data may be raw sample data from the second image, and the generation of the structural image data may involve overlaying the reference data with the raw sample data. This means that the sample data may be directly overlaid with the reference data without further preprocessing, e.g., without segmentation. This may simplify the generation of the structural image data.

[0021] In one example, the first image of the sample is indicative of a first biomarker and the second image of the sample is indicative of a second biomarker.

[0022] Examples provide a method comprising obtaining reference sample data indicative of a first image of a sample from a first channel and obtaining threshold data indicative of a plurality of thresholds for performing a distance transformation. The method further comprises generating reference data using at least two thresholds from the plurality of thresholds by applying the distance transformation to the reference sample data and obtaining sample data indicative of a second image of the sample from a second channel different from the first channel. The method further comprises generating structural image data indicative of a structure of the sample based on the reference data and the sample data.

[0023] Various examples of the present disclosure relate to a corresponding computer program having program code for performing the above method when the computer program is executed on a processor. Short description of the characters

[0024] Some examples of devices and / or methods are described below only by way of example and with reference to the accompanying figures, in which Fig. shows a schematic representation of an example of a device; Fig. shows a flowchart of an example of a method for generating structural image data; Fig. shows an example of a grayscale distance transformation flowchart; The Fig. show examples of mapping functions; Fig. 5 shows a flowchart of a method; and Fig. shows a schematic representation of a system that includes a microscope and a computer system. Detailed description

[0025] Various examples will now be described in more detail with reference to the accompanying drawings, some of which are illustrated. In the figures, the thicknesses of lines, layers, and / or regions may be exaggerated for clarity.

[0026] Fig. 1 shows a schematic representation of an example of a device 130. The device 130 comprises, as in Fig. 1, one or more processors 134 and one or more storage devices 136. Optionally, the device 130 also includes one or more interfaces 132. The one or more processors 134 are connected to the one or more storage devices 136 and to the optional one or more interfaces 132. In general, the functionality of the device 130 may be provided by the one or more processors 134 (e.g., to generate the structural image data), in conjunction with the one or more interfaces 132 (to exchange information, e.g., to obtain the reference pattern data), and / or with the one or more storage devices 136 (to store and / or retrieve information).

[0027] The device 130 is configured to obtain reference sample data indicative of a first image of a sample from a first channel. The reference sample data may be obtained by receiving or retrieving it from an optical imaging system and / or an external storage device. For example, the sample data may be obtained by receiving the sample data from an optical imaging system (e.g., via interface 132), by retrieving the sample data from a memory of an optical imaging system (e.g., via interface 132) or another external storage device, and / or by retrieving the sample data from a storage device 136 of the device 130, e.g., after the sample data has been written to the storage device 136 by an optical imaging system or by another system or processor, e.g., having an external storage device.

[0028] Image data or a dataset obtained by imaging a sample may contain information specific to a particular tissue type or molecular marker obtained by imaging techniques such as immunofluorescence or immunohistochemistry. For example, the sample may be a specimen from which image data was acquired using imaging techniques such as an optical imaging system. The image data may be a multidimensional dataset. That is, the acquired image data of the sample may include individual images or datasets representing different biological samples, e.g., proteins, and / or molecular targets, e.g., biomarkers. For example, the first image of the sample from the first channel may be part of image data obtained by imaging a sample.The acquired image data can be multiplexed to extract a channel or layer of information into a single image, e.g., the first image. That is, the first image of the sample from the first channel can be generated by multiplexing the image data. Multiplexing can involve the extraction of a single channel or spectral component from a multidimensional dataset, e.g., the image data, allowing for targeted analysis and interpretation of specific features within the data. For example, the first image can represent a specific biomarker portion of the sample. Multiplexing can enable the simultaneous visualization and analysis of multiple targets in the same sample, facilitating the comprehensive characterization and understanding of complex biological systems.For example, in hyperspectral imaging, where each pixel contains information across a range of wavelengths, multiplexing techniques can be used to extract and analyze the spectral signature of specific materials or substances in the sample, such as various biomarkers.

[0029] As described above, the device 130 can obtain the reference pattern data by receiving or retrieving it. Alternatively, the device 130 can also receive or retrieve image data and determine the reference pattern data based on the image data, e.g., by multiplexing. For example, the device 130 can receive the image data as raw data from an optical imaging sensor of an optical imaging system. Multiplexing the image data can be performed, for example, by the device 130 by dividing the raw data from an optical image sensor into multiple channels, resulting in a multiplexed image set. In this way, the device 130 can generate a multiplexed image set (based on the image data) or receive or retrieve a multiplexed image set. When the multiplexed image set is received or retrieved, an external processor, e.g., a control unit of an optical imaging system, has generated the multiplexed image set.

[0030] A multiplexed image set allows the isolation and analysis of individual spectral channels, allowing researchers to investigate specific features or properties within the image data. For example, with multiple biomarkers with overlapping spectra, individual spectral channels can be extracted from a comprehensive dataset, such as the image data. This approach facilitates the targeted analysis of each individual spectral signature of a biomarker, even in cases where their spectra overlap, thus enabling detailed investigation of each biomarker presence and their distribution within the sample.

[0031] In general, a biomarker can accumulate (i.e., label) not only in one structure (or spatial feature), but in multiple structures, such as blood vessels, mesenchymal cells, fibroblasts, or endothelial cells. Therefore, one channel cannot provide unique information about a specific structure. Nevertheless, the channel can be used to extract the desired information about a specific structure, e.g., by using reference data from a reference channel containing a biomarker that accumulates only in one structure. Each channel can provide unique information about the properties of the sample (that was acquired). Analyzing these channels together can enable a more comprehensive understanding of the sample under study. That is, two different biomarkers from two different channels can be used to create a spatial context image, e.g.,to improve the visibility of a structure.

[0032] The first channel can, for example, be a single channel of a multiplexed image set. The first image of the sample can be an image in which a biomarker has accumulated in only one structure of the sample. That is, the first image of the sample can be used as a reference for another channel containing a different image of the sample, e.g., a second image of the sample containing a biomarker accumulated in multiple structures. In this way, the first image of the sample can be used to highlight a specific structure of the multiple structures in the second image of the sample and generate a spatial context feature image. For example, a first biomarker can be accumulated in a first structure of the first image. A second biomarker can be enriched in multiple structures of the second image of the sample, including the first structure.In this case, the first image of the sample can be used to enhance the first structure in the second image (or suppress the other structures). In this way, the first image of the sample can be used to generate a special context feature image in which the first structure is enhanced. A distance transform can be used to generate the spatial context feature image.

[0033] To quantify the distance between objects labeled with two biomarkers from two different channels, both target and reference objects are segmented from their respective biomarker images or channels. A standard distance transformation can then be applied to a segmentation mask resulting from the segmentation of the reference marker image. For example, a spatial feature is determined based on a standard distance transformation, which requires a segmentation mask as input. Based on the segmentation mask, target objects are retained if they fall within the specified distance threshold, while those outside the specified range are removed. However, the segmentation process is very time-consuming and error-prone.No single segmentation method can consistently perform the segmentation process for a multiplexed biomarker image set, especially for images with complex structures, noisy backgrounds, or quality issues. It is the inventors' insight that the generation of a spatial contextual feature image can be improved by a segmentation-free process. That is, the device 130 is configured to perform a segmentation-free operation. Therefore, a segmentation mask may not be required. The operation is based on the reference pattern data (as described above), a distance transform, and pattern data from a channel of interest. The device 130 can avoid segmentation and directly quantify the proximity or distance between individual structures in multiplexed biomarker images.

[0034] To enhance or suppress biomarker expression signals characterized by different biomarkers, i.e., from different channels, a distance transformation can be used. Therefore, the device 130 can be configured to perform a distance transformation. Therefore, the device 130 is configured to receive threshold data indicating a plurality of thresholds for performing a distance transformation. A threshold is a predefined value used to divide a grayscale or color image, e.g., the first image, into two regions based on pixel intensity. Pixels with an intensity above the threshold are typically classified as foreground (or structure), while pixels below the threshold are classified as background.

[0035] The threshold data may be obtained by receiving or retrieving it from an optical imaging system and / or an external storage device. For example, the threshold data may be obtained by receiving the threshold data from an optical imaging system (e.g., via interface 132), by retrieving the threshold data from a memory of an optical imaging system (e.g., via interface 132) or another external storage device, and / or by retrieving the threshold data from a storage device 136 of the device 130, e.g., after the threshold data has been written to the storage device 136 by an optical imaging system or by another system or processor, e.g., having an external storage device.

[0036] Furthermore, the device 130 is configured to generate reference data by applying the distance transformation to the data of the reference sample. The reference data is generated using at least two thresholds from the plurality of thresholds. The reference data may be indicative of a distance-transformed first image of the sample. For example, the distance-transformed first image of the sample may be generated by averaging a plurality of distance-transformed first intermediate images of the sample, as described below. That is, the device 130 may use the threshold data to perform a distance transformation. The distance transformation may comprise a plurality of intermediate distance transformations to generate a plurality of intermediate distance-transformed first images. Furthermore, the distance transformation may comprise averaging over a plurality of intermediate distance transformations, i.e.averaging across the plurality of intermediate distance transformed images. Each intermediate distance transform may be based on a threshold from the plurality of thresholds. The threshold data may be used, for example, to generate binary data (as described in more detail below) or to identify regions of interest within the first image. The distance transformation may be performed based on the binary data or the regions of interest in the first image. The distance-transformed first image may include information about a transformed first image, which may typically include the distances from each pixel to a particular structure (or spatial feature) or region of interest in the first image.

[0037] The distance transformation can be applied directly to the reference sample data to generate the distance-transformed first image. This eliminates the need to segment the first image. This means that the reference sample data can be used as a reference to generate a spatial context feature image without segmentation. Therefore, based on the reference sample data, a structure in another image of the sample can be enhanced or suppressed.

[0038] Thus, the device 130 is further configured to receive sample data indicating a second image of the sample from a second channel, different from the first channel. In principle, the sample data may be acquired in the same way as the reference sample data. The second channel may, for example, be a single channel of a multiplexed image. The second image of the sample from the second channel may, for example, be part of image data acquired by acquiring an image of a sample. The acquired image data may be multiplexed to generate the second image (and optionally, simultaneously, the first image). That is, the second image of the sample from the second channel may be generated by multiplexing the image data. For example, the second image or second channel may be part of the same multiplexed image set as the first image of the first channel.The multiplexed image set can be generated based on image data acquired by imaging a sample. That is, the sample data can be acquired simultaneously with the reference sample data, e.g., by receiving a multiplexed image set comprising the first channel and the second channel. Alternatively, the sample data can also be acquired separately from the reference sample data. For example, the device 130 can determine the sample data separately from the reference sample data based on image data.

[0039] A structure within the second image of the sample can be enhanced by using the data from the reference sample. This can enable the generation of an image of the sample in which the visibility of a desired structure is enhanced. Thus, the device 130 is configured to generate structural image data indicative of a structure of the sample. The structural image data is generated based on the reference data and sample data. For example, the reference data and the sample data can be multiplied to generate structural image data. The structural image data can contain information about a spatially contextual feature image showing the structure of the sample. When generating the structural image data, target signals (i.e., pixel values ​​in a target image, namely the second image) can be enhanced if they are close to reference signals of the first image (i.e., pixel values ​​in the distance-transformed first image, e.g.,correspond to the first structure), while simultaneously suppressing them if they are far from the reference signals. In this way, a structure in the second image can be enhanced based on a structure, e.g., the same or a comparable structure, in the first image. In this way, the visibility of a structure in the sample can be improved.

[0040] In principle, a pixel value in the second image is also called a target signal because the second image is the target for determining or enhancing a structure. A pixel value in the distance-transformed first image (or the mapped distance-transformed first image, as described below) is also called a reference signal because the first image can be used as a reference for enhancing or suppressing the target signal. The pixel values ​​in the first image can come from a first biomarker. The pixel values ​​in the second image can come from a second biomarker that is different from the first biomarker. That is, the target signal can indicate at least one spatial feature (e.g., a structure) in which the second biomarker has accumulated. The reference signal can indicate at least one spatial feature in which the first biomarker has accumulated.Thus, reference signals (biomarkers) can be used to enhance or suppress target signals (biomarkers). A target signal or reference signal can originate from a specific biomarker in the sample.

[0041] In principle, any digital image can be used as the first or second image. The first and / or second images can, for example, be a fluorescence image and / or a confocal image obtained by capturing a sample and multiplexing the acquired image data.

[0042] Thus, the device 130 may be configured to generate a spatial contextual feature image based on a reference image (e.g., the first image) containing information about a first biomarker (also referred to as a reference marker) and a target image (e.g., the second image) containing information about a second biomarker (also referred to as a target market) that differs from the first biomarker. That is, the device 130 may employ a segmentation-free method for generating a spatial contextual feature image.

[0043] A spatial contextual feature image can be an image that represents a spatial relationship within a specific sample or a dataset obtained by imaging a sample. It does not directly depict the physical appearance of objects, but rather encodes information about the spatial arrangement, layout, or relationships between different elements within the sample. Therefore, the structure of the sample can be integrated into, or be part of, the spatial contextual feature image. That is, the process of applying a distance transformation and generating the structural image data based on the reference data and the sample data can be a segmentation-free operation.

[0044] The structural image data generated by device 130 may be considered non-segmentation-based. Segmentation typically involves dividing an image into distinct regions or components based on certain criteria such as intensity, color, or texture similarity. Distance transformation does not divide the image into distinct regions, but rather calculates the distance of each pixel to a nearest boundary or edge. That is, distance transformation quantifies the spatial relationships within the image rather than segmenting it into distinct regions. This distinguishes distance transformation from conventional segmentation methods, which divide an image based on certain criteria to delineate objects.

[0045] In one example, generating the reference data may include applying a mapping function to a distance-transformed first image of the sample. The mapping function may be applied to individual pixels of the distance-transformed first image. For example, the mapping function may be applied to a plurality of pixels, e.g., subsequently to all pixels or each pixel, of the distance-transformed first image to convert short-distance values ​​to high values ​​and long-distance values ​​to low values. For example, the mapping function may be applied to the distance-transformed first image to generate a mapped distance-transformed first image. The mapping function may control how target signals, i.e., pixel values ​​in the second image, may be enhanced or suppressed based on the distance of the reference signal, i.e., the pixel values ​​in the mapped distance-transformed first image (see also Fig. 4). The mapping function can be, for example, a Gaussian function. Alternatively, the mapping function can be any function.

[0046] In one example, the mapping function is a monotonically decreasing function, a monotonically increasing function, and / or a bandpass function. For example, the mapping function may be a monotonically decreasing function such as a Cauchy distribution function, a T-distribution function, or an inverse sigmoid function to enhance target signals close to the reference signals. For example, the mapping function may be a monotonically increasing function such as an exponential function, a quadratic function, a sigmoid function, or a Cauchy loss function to enhance target signals far from the reference signals. For example, the mapping function may be a bandpass function such as a bandpass function or an inverse bandpass function to enhance target signals that are within a certain distance range relative to the reference signals.

[0047] In one example, the distance transformation may be a grayscale distance transformation. The grayscale distance transformation may be applied to the first image to generate a distance-transformed first image in which pixels with lower intensity values ​​indicate greater proximity to reference signals (e.g., from reference markers).

[0048] In one example, one of the plurality of thresholds for performing the distance transformation may be an intensity threshold for retaining or excluding pixels from performing the distance transformation. Setting an intensity threshold may allow pixels to be selectively included or excluded from the distance transformation process based on their intensity values. This may allow the distance transformation to be focused on specific regions or features of interest within the first image. Selective means selecting or singling out certain elements or components from a larger set, while disregarding others based on certain criteria or preferences. It involves making a conscious decision to purposefully include or exclude certain elements or actions.

[0049] Furthermore, the variety of thresholds can enable noise reduction. For example, intensity thresholds can help filter out noisy or irrelevant pixels from the distance transformation. By excluding low-intensity pixels, which may correspond to noise or background noise, the accuracy and reliability of the distance transformation and thus of the structural image data can be improved.

[0050] Furthermore, the large number of thresholds can improve computational performance. Thresholding before performing the distance transformation can reduce the number of pixels involved in the calculation, resulting in improved computational efficiency. This allows for more efficient analysis of complex images.

[0051] In one example, device 130 may be configured to generate intermediate binary data (or binary images) indicating a plurality of binary-like representations or a plurality of binary masks for performing the distance transformation. The intermediate binary data is generated based on the reference pattern data and the threshold data. Further, generating the reference data may include applying the distance transformation to the binary data to generate a plurality of distance-transformed first intermediate images. That is, the output images, i.e., the binary data or binary images generated by applying a plurality of thresholds, may subsequently be used to apply the distance transformation, i.e., a plurality of distance transformations.The plurality of distance transformations generated based on the plurality of binary-like representations or a plurality of binary masks can make it possible to provide an averaged distance transformation image. That is, the plurality of distance transformations can be used to generate a plurality of intermediate distance-transformed first images. The plurality of intermediate distance-transformed first images can be used to generate an average distance-transformed first image. Thus, the distance transformation can be used to generate the average distance-transformed first image. That is, the distance-transformed first image described above can also be referred to as an average distance-transformed first image because it is generated based on multiple thresholds.

[0052] In one example, device 130 may be configured to generate the (average) distance-transformed first image by averaging the distance-transformed first intermediate images. For example, each threshold value from the plurality of threshold values ​​may be used to generate a distance-transformed first intermediate image. Thus, the (average) distance-transformed first image may be generated by averaging a plurality of distance-transformed first intermediate images.

[0053] Binary data, or a binary image, refers to an image in which each pixel can only take one of two possible values, usually 0 and 1, representing the background and foreground, respectively. The binary image can be a binary-like representation or a binary mask. A binary mask is a binary image in which each pixel is classified as either foreground (structure) or background. A binary-like representation is a binary image that resembles or behaves like a binary mask, regardless of strict adherence to binary conventions.

[0054] The distance transformation can calculate the distance of each pixel in the binary image to the nearest boundary or edge using a distance metric such as Euclidean distance. The output (data) of the distance transformation process is a distance-transformed image. In this distance-transformed image, the intensity value of each pixel represents its distance to the nearest boundary or edge in the original binary image. Pixels closer to the boundary have lower intensity values, while pixels farther away have higher intensity values.

[0055] With distance transformation, the distances between each pixel and the nearest boundary in the binary image are measured, rather than segmenting the image into different regions. The output data of the distance transformation, i.e., the reference data, is therefore a distance-transformed first image, where each pixel value represents the distance to the nearest boundary in the binary image, rather than a segmented image with distinct regions. This means that no division into regions is performed. While distance transformation provides valuable information about the spatial relationships within the image, it does not divide the image into individual regions or components based on similarity criteria, as is the case with segmentation methods. Therefore, the device 130 can avoid segmentation and directly measure the proximity or distance between individual objects in multiplexed biomarker images, e.g.,the first image and the second image.

[0056] In one example, device 130 may be further configured to generate overlay image data indicating the structure of the image. The overlay image data is generated based on the structure image data and the reference data. For example, the overlay image data may be generated by multiplying or dividing the distance-transformed first image or the mapped distance-transformed first image with or by the second image. In this way, the target signals of the second image are enhanced when they are near the reference signals and suppressed when they are far from the reference signals.

[0057] In one example, the first channel and the second channel may be part of a multiplexed image set. That is, the reference sample data and the sample data may be acquired simultaneously. For example, the device 130 may generate the structural image data based on multiplexed images of a sample. For example, the device 130 may process only two images from a multiplexed image set. For example, the device 130 may retrieve a set of multiplexed images and select the first image and the second image from the retrieved set of multiplexed images.

[0058] In one example, the generation of the structural image data may be a segmentation-free process. A segmentation-free process may provide spatial information without the need for explicit segmentation, thereby simplifying the analysis process and reducing computational complexity. This approach may be particularly advantageous in scenarios where conventional segmentation methods are difficult or impractical due to complex structures, noisy backgrounds, and / or overlapping structures. By directly quantifying the distances between individual pixels and the relevant features or boundaries, device 130 may enable efficient analysis and interpretation of the spatial relationships within the second image. In this way, a multiplexed image, e.g., the multiplexed image set, may be analyzed in an enhanced manner.

[0059] In one example, the generation of the structural image data may retain an unsegmented feature of the first image of the sample and the second image of the sample. That is, the structural image data may be generated without segmentation. As described above, this may enable the analysis of images with complex structures, noisy backgrounds, and / or overlapping structures.

[0060] In one example, generating the structural image data may include highlighting a portion of the second image corresponding to a portion of the reference data with a short distance and / or attenuating (or suppressing) a portion of the second image corresponding to a portion of the reference data with a long distance. Thus, a distance transformation may be used to generate structural image data. Using the distance transformation may make it possible to avoid segmentation of the first image and / or the second image. For example, when studying the microenvironment of T cells, it is advantageous to enhance targeting signals from target markers that are close to T cells (e.g., the second image) and labeled with a T cell-specific reference marker (e.g., the first image), while suppressing targeting signals that are distant from the T cells.In this case, the first image would contain reference signals indicating the spatial features of the T cells and could be used to enhance the targeting signals of the second signals corresponding to the T cells. The targeting signals could be indicative of the T cells and other structures. Using the reference signal may make it possible to highlight the part of the second image corresponding to the labeled part of the first image resulting from the reference marker, i.e., the T cells. That is, the part of the second image indicating the T cells could be highlighted using the reference signal indicating only the T cells.

[0061] In one example, generating the structural image data may include highlighting a spatial feature in the sample image data that corresponds to a structure represented by distance values ​​in the reference data. The spatial feature, e.g., a structure, may be highlighted based on the representation of the structure and the first image, for example. For example, the identification of blood vessels in multiplexed images may be performed using device 130. Blood vessels may be labeled with various biomarkers. While some of these biomarkers (e.g., the reference marker) may exclusively label vessels, other biomarkers (e.g., the target marker) could also label additional cell types, such as muscle cells. For example, the first image may be from a reference marker that accumulates only in the blood vessels, and the second image may be from a target marker that accumulates in the blood vessels and other structures.In this case, the device 130 can preprocess the non-vessel-specific biomarker images, i.e., the second image, while retaining only the signals that closely match the signals of the vessel-specific markers, i.e., the first image. This preprocessing can be performed by the device 130 without segmentation.

[0062] In one example, the reference pattern data may be raw reference pattern data of the first image of the sample, and the distance transformation to generate the reference data is applied to the raw reference pattern data. That is, the distance transformation may be applied directly to the reference sample data without further preprocessing. The reference sample data may be retrieved, for example, from an external storage device. The raw reference sample data may be generated by multiplexing sensor data as described above. That is, the raw reference data may be raw data generated by multiplexing sensor data, e.g., from an optical image sensor of an optical imaging system. In one example, the sample data may be raw sample data of the second image, and generating the structural image data may include overlaying the reference data with the raw sample data.This means that the sample data can be directly overlaid with the reference data without any further preprocessing, e.g., without segmentation.

[0063] In one example, the first image of the sample is indicative of a first biomarker and the second image of the sample is indicative of a second biomarker.

[0064] The device 130 can be external to an optical imaging system. Alternatively, the device 130 can also be part of an optical imaging system, e.g., as a control unit of the optical imaging system.

[0065] As in Fig. 1, the optional interface(s) 132 are coupled to the respective one or more processors 134 on the device 130. In examples, the one or more processors 134 may be implemented by one or more processing units, one or more processing devices, any means of processing, such as a processor, a computer, or a programmable hardware component operable with appropriately adapted software. Likewise, the described functions of the one or more processors 134 may also be implemented in software that is then executed on one or more programmable hardware components. Such hardware components may include a general-purpose processor, a digital signal processor (DSP), a microcontroller, etc.The one or more processors 134 are capable of controlling the one or more interfaces 132 such that any data transmission that occurs over the one or more interfaces 132 and / or any interaction that may involve the one or more interfaces 132 may be controlled by the one or more processors 134.

[0066] In one embodiment, device 130 may include a memory, e.g., one or more storage media 136, and at least one or more processors 134 operably coupled to the memory and configured to perform the method described below.

[0067] In examples, the one or more interfaces 132 may correspond to any means for acquiring, receiving, transmitting, or providing analog or digital signals or information, e.g., any terminal, contact, pin, register, input terminal, output terminal, conductor, trace, etc., that enables the provision or acquisition of a signal or information. The one or more interfaces 132 may be wireless or wired and may be configured to communicate with other internal or external components, e.g., to transmit or receive signals or information.

[0068] The device 130 may be a computer, a processor, a controller, a field-programmable logic array (FPLA), a field-programmable gate array (FPGA), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), an integrated circuit (IC), or a system-on-a-chip (SoC).

[0069] Further details and aspects are mentioned in connection with the examples described below. Fig. 1 may include one or more optional or additional features corresponding to one or more aspects related to the proposed concept or one or more examples described below (e.g. Fig. 2 - 6).

[0070] Fig. 2 shows a flowchart of an example of a method 200 for generating structural image data 280, e.g., a spatial context feature image 280. The method 200 may be performed by a device as described above, e.g., with reference to Fig. 1.

[0071] The method 200 may include obtaining reference pattern data, e.g., a reference marker image 210 (e.g., the first image). At 220, a distance transformation, e.g., a grayscale distance transformation, may first be applied to the reference marker image 210, which may generate a distance-transformed image 230. Pixels in the distance-transformed image 230 with lower intensity values ​​indicate greater proximity to the reference marker signals. The distance transformation may be performed as described above, e.g., using multiple thresholds and an optional binary mask.

[0072] At 240, a mapping function can be applied to individual pixels within the distance-transformed image 230. By applying the mapping function, short distance values ​​can be converted to high values ​​and long distance values ​​to low values. The mapping function can be, for example, a Gaussian function with a mean of zero. The output data can be a mapped distance-transformed image 250, for example, a Gaussian-mapped image.

[0073] The method may further include obtaining sample data, e.g., a target marker image 260 (e.g., a second image). At 270, an overlay of the target marker image 260 and the mapped distance-transformed image may be generated. The overlay results in a spatial contextual feature image 280. If a Gaussian function was used as the mapping function, the target signals are enhanced when they are near the reference signals and suppressed when they are far from the reference signals. In this way, a spatial contextual feature image 280 can be generated without separation.

[0074] Further details and aspects are mentioned in connection with the examples described above and / or below. Fig. The example shown in Figure 2 may comprise one or more optional or additional features corresponding to one or more aspects related to the proposed concept or one or more of the above (e.g. Fig. 1) and / or below (e.g. Fig. 3 - 6) are mentioned.

[0075] Fig. shows an example of a flowchart of the grayscale distance transformation 300. The grayscale distance transformation 300 may be performed by a device as described above, e.g., with reference to Fig. 1.

[0076] The method 300 may include obtaining reference sample data, e.g., a reference marker image 210 (e.g., the first image). At 320 (T1 -T n ) can be a sequence of several thresholds, denoted as T1 < T2 < T3 < ... < T n , are applied successively to the reference marker image 210. The application of the multiple threshold values ​​T1 - Tn on the reference marker image 210 generates a series of binary masks 3301, 3302, 3303, etc., up to 330 .n

[0077] At 340, a distance transformation, e.g., a regular distance transformation, is applied to each of the binary masks 3301,..., 330 n The application of the distance transformation results in the generation of a set of distance-transformed images 3501, 3502, 3503, etc., up to 350 .n

[0078] At 360, an averaging process can be performed for all distance-transformed images 3501,..., 350 n The averaging process produces an (averaged) distance-transformed grayscale image 370. The distance-transformed image 370 can, for example, be Fig. 2 shown distance-transformed image (reference number 220).

[0079] Further details and aspects are mentioned in connection with the examples described above and / or below. Fig. 3 may comprise one or more optional or additional features corresponding to one or more aspects related to the proposed concept or one or more of the above (e.g. Fig. 1 - 2) and / or below (e.g. Fig. 4 - 6) are mentioned.

[0080] The Fig. show examples of mapping functions. The Fig. show various monotonically decreasing functions 410, 420, 430 that can be used as mapping functions. A monotonically decreasing function 410, 420, 430 amplifies the target signals that are close to the reference signals and suppresses target signals that are far from the reference signals. For example, a Cauchy distribution function 410 ( Fig. , a T-distribution function 420 ( Fig. , also called Student's t-distribution function) or an inverse sigmoid function 430 ( Fig. ) be used.

[0081] Fig. show various increasing functions 440, 450, 460, 470 that can be used as a mapping function. A monotonically increasing function 440, 450, 460, 470 suppresses the target signals that are close to the reference signals and amplifies the target signals that are far from the reference signals. For example, an exponential function 440 ( Fig. 4d), a quadratic function 450 ( Fig. 4e), a sigmoid function 460 ( Fig. 4f) or a Cauchy loss function 470 ( Fig. 4g) may be used.

[0082] The Fig. show a bandpass function 480 ( Fig. and an inverse bandpass function 490 ( Fig. ), which can be used as a mapping function. A bandpass function 480 or an inverse bandpass function 490 amplifies the target signals that lie within a certain distance range from the reference signals, while suppressing target signals that lie outside this range.

[0083] Further details and aspects are mentioned in connection with the examples described above and / or below. Fig. 4 may include one or more optional or additional features corresponding to one or more aspects related to the proposed concept or one or more of the above (e.g. Fig. 1 - 3) and / or below (e.g. Fig. 5 - 6) are mentioned.

[0084] Fig. 5 shows a flowchart of a method 500. The method 500 includes obtaining 510 reference sample data indicative of a first image of a sample from a first channel and obtaining 520 threshold data indicative of a plurality of threshold values ​​for performing a distance transformation. The method 500 further includes generating 530 reference data using at least two threshold values ​​from the plurality of threshold values ​​by applying the distance transformation to the reference sample data and obtaining 540 sample data indicative of a second image of the sample from a second channel different from the first channel. The method 500 further includes generating 550 structural image data indicative of a structure of the sample based on the reference data and the sample data. The method 500 can be performed using a device as described above, e.g., with reference to Fig. 1.

[0085] Further details and aspects are mentioned in connection with the examples described above and / or below. Fig. The example shown in Figure 5 may comprise one or more optional or additional features corresponding to one or more aspects related to the proposed concept or one or more of the above (e.g. Fig. 1 - 4) and / or below (e.g. Fig. 6) are mentioned.

[0086] Some embodiments relate to a microscope with a device as used in connection with Fig. 1. Alternatively, a microscope or an optical imaging system may be communicatively connected to a device as used in connection with Fig. 1 described. Fig. Figure 6 shows a schematic representation of a system 600, e.g., an optical imaging system, configured to perform a method described herein, e.g., with reference to one or more of the Fig. 2 or Fig. 5. The system 600 includes a microscope 610 and a computer system 620. The microscope may include the device described above, e.g., with reference to Fig. 1. The microscope 610 is configured to capture images and is connected to the computer system 620. The computer system 620 is configured to execute at least part of a method described herein. The computer system 620 may be configured to execute a machine learning algorithm. The computer system 620 and the microscope 610 may be separate units, but may also be integrated into a common housing. The computer system 620 may be part of a central processing system of the microscope 610 and / or the computer system 620 may be part of a subcomponent of the microscope 610, such as a sensor, an actuator, a camera, or an illumination unit, etc., of the microscope 610.

[0087] Computer system 620 may be a local computing device (e.g., a personal computer, laptop, tablet computer, or mobile phone) with one or more processors and one or more storage devices, or a distributed computing system (e.g., a cloud computing system with one or more processors and one or more storage devices distributed at different locations, e.g., at a local client and / or one or more remote server farms and / or data centers). Computer system 620 may include any circuitry or combination of circuitry. In one embodiment, computer system 620 may include one or more processors, which may be of any type. As used herein, the term "processor" may refer to any type of computing circuitry, such as:a microprocessor, a microcontroller, a CISC (Complex Instruction Set Computing) microprocessor, a RISC (Reduced Instruction Set Computing) microprocessor, a VLIW (Very Long Instruction Word) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a FPGA (Field Programmable Gate Array), e.g., of a microscope or a microscope component (e.g., camera), or any other type of processor or processing circuit. Other types of circuitry that may be included in computer system 620 may be a custom circuit, an application-specific integrated circuit (ASIC), or the like, such as one or more circuits (e.g., a communications circuit) for use in wireless devices such as cellular phones, tablet computers, laptop computers, two-way radios, and similar electronic systems.Computer system 620 may include one or more storage devices, which may include one or more storage elements suitable for the particular application, such as main memory in the form of random access memory (RAM), one or more hard disks, and / or one or more drives that handle removable media such as compact disks (CDs), flash memory cards, digital video disks (DVDs), and the like. Computer system 620 may also include a display device, one or more speakers, and a keyboard and / or a control device, which may include a mouse, a trackball, a touchscreen, a voice recognition device, or other device that enables a system user to input information to and receive information from computer system 620.

[0088] Further details and aspects are mentioned in connection with the examples described above. Fig. 6 may include one or more optional or additional features corresponding to one or more aspects related to the proposed concept or one or more examples described above (e.g. Fig. 1 - 5) were mentioned.

[0089] Some or all of the method steps may be performed by (or using) a hardware device, such as a processor, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, some or more of the key method steps may be performed by such a device.

[0090] Depending on specific implementation requirements, embodiments of the invention may be implemented in hardware or software. The implementation may be performed using a non-transferable storage medium such as a digital storage medium, for example, a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or FLASH memory, on which electronically readable control signals are stored that cooperate (or can cooperate) with a programmable computer system to perform the respective method. Therefore, the digital storage medium may be computer-readable.

[0091] Some embodiments of the invention comprise a data carrier with electronically readable control signals capable of cooperating with a programmable computer system so that one of the methods described herein is carried out.

[0092] In general, embodiments of the present invention can be implemented as a computer program product with program code, wherein the program code serves to execute one of the methods when the computer program product is run on a computer. The program code can, for example, be stored on a machine-readable medium.

[0093] Other embodiments include the computer program for performing one of the methods described herein stored on a machine-readable medium.

[0094] In other words, one embodiment of the present invention is therefore a computer program having a program code for carrying out one of the methods described here when the computer program runs on a computer.

[0095] A further embodiment of the present invention is therefore a storage medium (or a data carrier or a computer-readable medium) on which the computer program for performing one of the methods described herein is stored when executed by a processor. The data carrier, the digital storage medium, or the recorded medium is typically tangible and / or non-transferable. A further embodiment of the present invention is a device as described herein, comprising a processor and the storage medium.

[0096] A further embodiment of the invention is therefore a data stream or a sequence of signals representing the computer program for carrying out one of the methods described here. The data stream or signal sequence can, for example, be configured to be transmitted via a data communication connection, e.g., via the Internet.

[0097] Another embodiment includes a processing means, e.g., a computer or programmable logic device, configured or adapted to perform any of the methods described herein.

[0098] A further embodiment comprises a computer on which the computer program for carrying out one of the methods described here is installed.

[0099] A further embodiment of the invention comprises a device or system configured to transmit a computer program for performing one of the methods described herein to a recipient (e.g., electronically or optically). The recipient may be, for example, a computer, a mobile device, a storage device, or the like. The device or system may, for example, comprise a file server for transmitting the computer program to the recipient.

[0100] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field-programmable gate array may cooperate with a microprocessor to perform any of the methods described herein. In general, the methods are preferably performed by any hardware device.

[0101] If some aspects are described with respect to a device or system, these aspects should also be understood as a description of the corresponding method, and vice versa. For example, a block, device, or functional aspect of the device or system may correspond to a feature, such as a method step, of the corresponding method. Accordingly, aspects described with respect to a method should also be understood as a description of a corresponding block, element, property, or functional feature of a corresponding device or system.

[0102] The following claims are hereby incorporated into the detailed description, each claim being understood to stand on its own as a separate example. It should also be noted that although in the claims a dependent claim refers to a particular combination with one or more other claims, other examples may include a combination of the dependent claim with the subject matter of another dependent or independent claim. Such combinations are hereby expressly contemplated unless it is stated in a particular case that a particular combination is not intended. Furthermore, the features of a claim should also be included in any other independent claim, even if that claim is not directly defined as dependent on that other independent claim.

[0103] The aspects and features described with respect to a particular one of the preceding examples may also be combined with one or more of the further examples to replace an identical or similar feature of that further example or to additionally introduce the features into the further example. List of reference signs 130 devices 132 Interface 134 processor 200 methods for generating structural image data 210 Reference marker image 220 Application of a distance transformation 230 distance-transformed image 240 Application of a mapping function 250 distance-transformed image shown 260 Target Marking Image 270 Applying an overlay 280 spatial-contextual feature image 300 grayscale distance transformation methods 320 (T1 -T n) using a sequence of multiple thresholds T1 -T n Thresholds 3301,..., 330 n Binary masks 340 Application of distance transformation 3501, 3502, 3503 distance-transformed images 360 Calculating an average 370 distance-transformed image 410 Cauchy distribution function 420 T-distribution function 430 inverse sigmoid function 440 Exponential function 450 quadratic function 460 a sigmoidal function 470 Cauchy loss function 480 bandpass function 490 inverse bandpass functions 500 procedures 510 Obtaining data on reference samples 520 Acquisition of threshold data 530 Generation of reference data 540 Obtaining sample data 550 Generation of structural image data 600 system 610 Microscope 620 computer system QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] US 2018 / 0 240 239 A1

[0003] Cited non-patent literature

[0000] ZHANG, Chao, [et al.]: Clustered nuclei splitting via curvature information and gray-scale distance transform

[0004]

Claims

[1] A device (130) comprising one or more processors (134) and one or more memory devices (136), the device (130) being configured to: obtaining reference sample data indicating a first image of a sample (110) from a first channel; obtain threshold data specifying a plurality of threshold values ​​for performing a distance transformation; Generating reference data using at least two thresholds from the plurality of thresholds by applying the distance transformation to the reference sample data; Obtaining sample data indicating a second image of the sample (110) from a second channel different from the first channel; and to generate structural image data based on the reference data and the sample data, which indicate a structure of the sample (110). [2] The apparatus (130) of claim 1, wherein generating the reference data comprises applying a mapping function to a distance-transformed first image of the sample. [3] Device (130) according to one of the preceding claims, wherein the mapping function is at least one of a monotonically decreasing function, a monotonically increasing function or a bandpass function. [4] Apparatus (130) according to any one of the preceding claims, wherein the distance transformation is a grayscale distance transformation. [5] Apparatus (130) according to any one of the preceding claims, wherein the apparatus (130) is configured to generate overlay image data indicating the structure of the image based on the structure image data and the reference data. [6] The apparatus (130) of any preceding claim, wherein one of the plurality of thresholds for performing the distance transformation is an intensity threshold for retaining or excluding pixels for performing the distance transformation. [7] Apparatus (130) according to any one of the preceding claims, wherein the apparatus (130) is configured to generate intermediate binary data indicative of a plurality of binary-like representations or a plurality of binary masks for performing the distance transformation based on the reference pattern data and the threshold data, and wherein generating the reference data comprises applying the distance transformation to the binary data to generate a plurality of distance-transformed first images. [8] The apparatus (130) of any preceding claim, wherein the first and second channels are part of a multiplexed image set. [9] Apparatus (130) according to any one of the preceding claims, wherein the generation of the structural image data is a segmentation-free process. [10] The apparatus (130) of any preceding claim, wherein generating the structural image data maintains an unsegmented feature of the first image of the sample and the second image of the sample. [11] The apparatus (130) of any preceding claim, wherein generating the structural image data comprises highlighting a portion of the second image corresponding to a portion of the reference data having a short distance and / or attenuating a portion of the second image corresponding to a portion of the reference data having a long distance. [12] Apparatus (130) according to any one of the preceding claims, wherein generating the structural image data comprises highlighting a spatial feature in the sample image data that corresponds to a structure represented by distance values ​​in the reference data. [13] Apparatus (130) according to any one of the preceding claims, wherein the reference pattern data is raw reference pattern data of the first image of the sample and the distance transformation is applied to the raw reference pattern data to generate the reference data. [14] The apparatus (130) of claim 13, wherein the scan data is raw data of the second image, and generating the structural image data comprises overlaying the reference data with the raw data of the sample. [15] The device (130) of any preceding claim, wherein the first image of the sample is indicative of a first biomarker and the second image of the sample is indicative of a second biomarker. [16] Optical imaging system (600) comprising a device (130) according to any one of the preceding claims. [17] A method (500) for an optical imaging system comprising: Obtaining (510) reference sample data indicative of a first image of a sample from a first channel; Obtaining (520) threshold data indicative of a plurality of thresholds for performing a distance transformation; generating (530) reference data using at least two thresholds from the plurality of thresholds by applying the distance transformation to the reference sample data; Obtaining (540) sample data indicating a second image of the sample from a second channel different from the first channel; and Generating (550), based on the reference data and the sample data, structural image data indicating a structure of the sample. [18] Computer program with a program code for carrying out the method (500) according to claim 17, when the computer program is executed on a processor.

Images