A microscope system and a method for improving microscope image quality using luminance value transformation

Abstract

In order to improve the image quality of the microscope image, a luminance value transformation (T) is applied to the microscope image (20). The parameter values of the luminance value transformation (T) depend on context information (K) of the microscope image (20), which at least specifies a recording characteristic and / or a structure of the presentation, and / or such that a difference between the microscope image (20) and a reference image (20R) is reduced, wherein the reference image (20R) is determined from the microscope image (20) or the associated context information (K).

Description

Technical Field

[0001] This invention relates to a microscope system and a computer-based method for improving the quality of microscope images. Background Technology

[0002] Microscopic images are typically displayed on a screen as live images immediately after capture or during observation for the user to view. In typical microscope control and display software, users can change display parameters to make structures in the microscope image more clearly visible. For example, the contrast, brightness, or gamma value of the image display can be modified to improve the visibility of the captured structures. Depending on the display software, display parameters can be changed via display curves, which graphically show the mapping between the brightness values ​​of the microscope image and the new brightness values. This allows setting, for example, the gamma value, or more specifically, setting a minimum brightness value (displayed as black) and a maximum brightness value (displayed as white).

[0003] However, manually adjusting display parameters is both time-consuming and error-prone. Based on user experience, only limited effects can be achieved, and accurate reproduction is often impossible. This makes comparing multiple images particularly difficult. Even automatic adjustments based on simple criteria or heuristics often fail to provide satisfactory results, requiring subsequent corrections, which in turn require expert knowledge. For example, grayscale / hue stretching can be performed automatically, moving the darkest pixel values ​​in the original image to black and the brightest pixel values ​​to white. Even if the brightness values ​​in the original image only cover a portion of the possible values ​​(e.g., values ​​from 10 to 220), it utilizes the entire available brightness range from a minimum (e.g., zero) to a maximum (e.g., 255). This can improve contrast, but is usually far from ideal. Ideal display parameters vary depending on factors such as the overall frequency distribution of pixel values, recording techniques (e.g., different processing methods for phase-contrast and fluorescence images), the displayed samples, image noise, and unwanted structures. For example, in the case of crosstalk in a fluorescence image, the corresponding structures should be less noticeable, not more noticeable. This cannot be achieved through simple automatic contrast enhancement or similar methods.

[0004] Therefore, simple normalization methods, such as minimum-maximum normalization or percentile normalization, cannot provide satisfactory results and require manual readjustment. The optimal settings for the display curve largely depend on the application scenario. For example, to eliminate hammer effect artifacts, the lower range of the luminance value distribution must be truncated more strongly.

[0005] The following article describes a method for improving the contrast of microscope images through histogram equalization, and can be used as background information: Sri Hartati et al., “Digital Microscopy and Its Image Processing Tools”, TELKOMNIKA, Vol. 9, No. 3, December 2011, pp. 565-574, ISSN: 1693-693; Marina E. Plissiti et al., “Segmentation of Overlapping Nuclei Using Spatial Adaptive Active Physical Models”, IEEE Transactions on Image Processing, Vol. 21, No. 11, November 2012.

[0006] Histogram equalization refers to distributing the frequency of different pixel values ​​more evenly to improve contrast. Adaptive histogram equalization or contrast-limited adaptive histogram equalization (CLAHE) can also be used to improve display quality; see the following for details: https: / / en.wikipedia.org / wiki / Adaptive_histogram_equalization In this process, the image is segmented into multiple regions (patches), each region undergoes histogram adjustment, and bilinear interpolation is performed at the boundaries between regions. To avoid over-amplifying image noise in low-contrast regions, contrast is limited. Therefore, when adjusting image brightness values ​​using histogram equalization, it is based solely on the statistical information of the corresponding image, meaning that application-specific optimized images cannot be obtained.

[0007] Machine learning image-to-image models can also improve display quality, for example, by increasing contrast, reducing noise, or eliminating artifacts. However, machine learning image-to-image models inherently carry the risk of creating illusions—that is, adding structures not present in the original image. Therefore, common image-to-image models struggle to improve display quality without distorting the original image. This risk can be mitigated through additional control mechanisms, but this increases workload and leaves residual risks. Furthermore, image-to-image models must process the full resolution of microscope images, resulting in relatively high computational costs. When using machine learning image-to-image models, manual adjustments are typically not possible, or are only possible within very limited scope. The negative consequences of image-to-image models include not only the addition of artifacts but also other distortions of the original data, such as significant alteration or elimination of image noise, or changes to image structure.

[0008] To better understand the technical background of this invention, various machine learning techniques will be discussed below.

[0009] For text data processing, Transformer-based language models (Large Language Models, LLMs) can be used, such as GPT and BERT, as described in the following literature: Brown, T. et al., “Language Models as Few-Shot Learners”, arXiv:2005.14165v4 [cs.CL] July 22, 2020 Devlin, J. et al., “BERT: Pre-training of a Deep Bidirectional Transformer for Language Understanding”, arXiv:1810.04805v2 [cs.CL] May 24, 2019 The Transformer technique described there can also be applied to image processing. A model called VisionTransformer (ViT), characterized by its use of Transformers for image processing, is described in the following literature: Dosovitskiy, A. et al., “A picture is worth a thousand words: Transformers for large-scale image recognition,” arXiv:2010.11929v2 [cs.CV] June 3, 2021 As an alternative to ViTs, Convolutional Neural Networks (CNNs) are also widely used in image processing.

[0010] Multimodal model learning maps text and related image data to a common multimodal feature space, where the outputs of the image encoder (e.g., ResNet or ViT) and the text encoder are projected onto this common feature space, as detailed in the following literature: Radford, A. et al., “Learning Transferable Visual Models from Natural Language Supervision”, arXiv:2103.00020v1 [cs.CV] February 26, 2021 The model can also be trained using reinforcement learning, for example, as described in the following literature: DeepSeek AI: “DeepSeek-R1: “Incentivizing Reasoning Ability in LLMs through Reinforcement Learning”, arXiv:2501.12948v1 [cs.CL] January 22, 2025 Summary of the Invention

[0011] One object of the present invention is to provide a microscope system and method that can improve the presentation of microscope images without producing hallucinations or other distortions in the displayed image content.

[0012] This objective is achieved by a microscope system and method having the features of the independent claims.

[0013] The present invention provides a computer-implemented method for improving the quality of microscope images, the method comprising: acquiring at least one microscope image; determining contextual information about the microscope image, wherein the contextual information relates at least to recording characteristics and / or the structure presented; determining a brightness value transformation based on at least the contextual information; and applying the brightness value transformation to the microscope image.

[0014] In another computer-implemented method for improving the image quality of microscope images according to the present invention, at least one microscope image is obtained. At least one reference image for the microscope image is determined, particularly using the microscope image and / or contextual information about the microscope image. A brightness value transformation of the microscope image is determined such that the difference between the microscope image and the reference image is reduced. The brightness value transformation of the microscope image is performed. As explained in more detail below, the difference between the microscope image and the reference image can be captured by pixels, is related to image style, or can be measured based on the characteristics of the frequency distribution of image brightness values.

[0015] The microscope system of this invention includes a microscope for image acquisition and a computing device for executing the computer-implemented method according to the invention.

[0016] The present invention also relates to a non-volatile, computer-readable data storage device comprising a computer program containing instructions that, when executed by a computer, cause the computer to perform at least one computer implementation method according to the present invention.

[0017] This invention enables the determination and use of luminance value transformations suitable for corresponding microscopic images. Luminance value transformations can improve display quality without producing illusions or other artifacts, as is common in typical image-to-image models. The luminance value transformations are based not only on statistical information but also on semantic information about the displayed or recorded content. Improving image quality specifically means increasing the contrast of the structure under study and / or reducing the contrast of unwanted artifacts.

[0018] Optional design Variations of the microscope system and method of the present invention are the subject of the dependent claims and are explained in the following description.

[0019] Different embodiments differ in the basis for determining the brightness value transformation. Specifically, the brightness value transformation can be set based on the microscope image, relevant contextual information, and / or the brightness value characteristics of the microscope image, as will be explained in detail below. Based on this data, at least one reference image or other reference data can be determined to define the parameter values ​​for the brightness value transformation.

[0020] Brightness value target data For brightness value transformation, target brightness value data, such as a brightness value histogram, can be determined. This is a typical feature of high-quality images obtained using microscope image acquisition methods. High-quality images have different brightness value histograms depending on the type of microscope image acquisition (e.g., phase contrast or fluorescence). Therefore, a brightness value histogram is determined for the type of microscope image acquired (e.g., an image retrieved from a database). Subsequently, brightness value transformation can be performed to make the brightness value histogram of the microscope image more similar to the brightness value histogram loaded from the database.

[0021] More generally, determining the brightness value transformation may involve identifying target brightness value data based on contextual information. The brightness value transformation is then set or determined to approximate the brightness value data of the microscope image with the target brightness value data. An approximation refers to reducing the difference between the microscope image brightness value data and the target brightness value data through the brightness value transformation.

[0022] Brightness value data and target brightness value data can, in principle, be any parameter describing the frequency distribution of brightness values ​​in the corresponding image. This can be the frequency distribution of the pixel values ​​themselves (called a histogram of brightness values), or, for example, the uniformity, entropy, or dispersion of that frequency distribution, and / or the minimum and maximum values ​​of the pixel value frequency distribution.

[0023] For example, target data for brightness values ​​can specify a target histogram (target frequency distribution) of the brightness values. Determining the brightness value transformation can make the brightness histogram of the microscope image approximate the target histogram.

[0024] In this context, a histogram is understood as the frequency distribution of pixel values ​​in an image, that is, indicating the frequency of each possible pixel value appearing in the image. Frequency distribution does not necessarily need to be represented graphically; besides a graphical histogram, the numerical values ​​of the frequency distribution can also be used. For an image consisting of X pixels, where the value of each pixel is between 0 and Y, the frequency distribution of each value between 0 and Y represents the frequency of that value appearing in the image. The sum of the frequencies of values ​​from 0 to Y is X.

[0025] The target brightness value data can also specify target histogram properties (attributes of the target frequency distribution), specifically the uniformity, entropy, or dispersion of the target brightness value frequency distribution. Brightness value transformation is determined by approximating the histogram characteristics of the microscope image's brightness values ​​to the target histogram characteristics. For example, the diffusion of the pixel value frequency distribution in a high-quality image equivalent to a microscope image is determined or loaded. The brightness value transformation is then set to make the diffusion of the microscope image's pixel value frequency distribution more similar to the determined / loaded diffusion.

[0026] Alternatively or additionally, luminance value target data can also indicate image statistical targets for luminance values, particularly the mean, standard deviation, contrast, and / or saturation of pixel values. The determination of luminance value transformations approximates the image statistics of the microscope image to the image statistical targets.

[0027] Brightness target data can be stored in a database containing various contextual information. A database query is performed using the contextual information of an existing microscope image to load the brightness target data applicable to that image. For example, the database can contain image statistics (mean, standard deviation, etc.) for different types of imaging (phase contrast, fluorescence, etc.). If image analysis or metadata indicates that the microscope image is a phase contrast image, then the brightness target data stored in the database for this purpose is loaded.

[0028] Machine learning models can also be used to infer target brightness values ​​from contextual information and / or microscope images, or to directly infer brightness value transformations. For example, machine learning models can be designed as classification or regression models. Machine learning models are particularly useful when a large amount of contextual information needs to be considered, where the database may not contain relevant data for every possible combination of different values ​​of the contextual information. Furthermore, learning models are a good option, especially when considering the microscope images themselves, a point that will be described in detail later.

[0029] Reference image In various embodiments, at least one reference image is determined for the microscope image. The reference image can be determined, in particular, from the microscope image or from contextual information about the microscope image.

[0030] To determine the brightness value transformation, one can compare microscope images with reference images, or compare properties derived from these images, particularly the frequency distribution of brightness values ​​or certain properties / parameters of the frequency distribution.

[0031] In some embodiments, the brightness value transformation makes the characteristics of the brightness value frequency distribution of the microscope image more closely resemble those of the brightness value frequency distribution of the reference image. Here, the attribute of the brightness value frequency distribution is also referred to as brightness value attribute or brightness value data.

[0032] In the above-described variant of the invention, the determination of the target luminance value data depends on contextual information, and the process may include: determining at least one reference image based on at least the contextual information, and determining the target luminance value data from at least one reference image. In this case, the characteristics of the luminance value frequency distribution of the reference image can be used as the target luminance value data.

[0033] Regardless of whether the determination of the brightness target data is based on contextual information, it can be specified that the brightness data of at least one reference image is used as the brightness target data, and the method of determining the brightness transformation makes the brightness data of the microscope image approximate the brightness target data.

[0034] In various variations, determining at least one reference image for a microscope image involves calculating the reference image from the microscope image using a learned model. Here, the reference image can be calculated in particular by utilizing an image-to-image mapping from the microscope image, or by utilizing a text-to-image model with contextual information about the microscope image, or by selecting from an image database.

[0035] For image-to-image models, they can be trained to compute a quality-enhanced output image (used as a reference image) from an input microscope image. Specifically, the model can be trained to make structures of interest more clearly visible and suppress background structures and / or improve contrast. For example, a training target image can be used during training to improve the visibility of structures of interest or suppress the visibility of interfering structures compared to the input image. For instance, the model can be trained as a generator in a GAN (Generative Adversarial Network) setup, and / or include UNet, diffusion models, CNNs (Convolutional Neural Networks), and / or ViTs (Visual Transformers). The reference image computed by the image-to-image model can be registered to the microscope image, ensuring that objects in the microscope image have the same image coordinates as objects in the reference image.

[0036] Image-to-image models can also contain multiple consecutive models, which can be trained together or separately. For example, an image-to-image model can include a Transformer encoder that computes a compressed display (feature vector) from a microscope image. A separately learned decoder generates an output image based on the feature vector.

[0037] Besides learning models, classic image-to-image transformation algorithms, such as the CLAHE algorithm (Contrast-Limited Adaptive Histogram Equalization), can also be used in principle. The generated images have higher display quality; for example, the structure of interest has good contrast, while background structures are not overly prominent. However, especially when using deep neural networks, there is no guarantee that new structures will not be generated. Nevertheless, the calculated image can serve as a reference image, and its brightness value distribution characteristics need to be mimicked. Modifying microscope images in this way can at least partially achieve the high image quality of the reference image without distorting the displayed content, because the brightness value transformation does not add any new structures to the microscope image.

[0038] A reference image can be registered with a microscope image. This means that the same structure / object lies at the same image coordinates in both images, or more generally, the relationship between the image coordinates in the two images is known for what is being depicted. A reference image can be registered to a microscope image, especially if the reference image is calculated from the microscope image via image-to-image mapping, particularly through a learned model.

[0039] Reference image generated by image generator Alternatively, at least one reference image for the microscope image can be determined by having an image generator calculate a reference image. The image generator receives the microscope image and / or contextual information about the microscope image as input and is designed to calculate the output image (i.e., the reference image).

[0040] For example, a machine learning model used as an image generator can be trained using a training target image (annotated) with desired image attributes or brightness values. During training, the model's output should match the training target image as closely as possible. This allows the model to learn to generate images with attributes that match the training target image. For example, the brightness value attribute could be the contrast of the structure of interest. The training target image can exhibit high-quality visibility or high-contrast target structures while suppressing or eliminating artifacts or unwanted image content (such as reflections).

[0041] An image generator can be tuned using a microscope image and / or contextual information, meaning it is designed to receive a microscope image and / or contextual information and use it to compute an output image (=reference image). For example, a machine learning language model (LLM, Large Language Model) can be used to describe a microscope image. This description provides contextual information input to the image generator. In this case, the image generator is designed as a text-to-image model. For example, an LLM can be used to determine one or more pieces of contextual information from a microscope image, such as image type and sample type, which will be described in more detail later. Besides LLM, a machine learning classification model or encoder can be used to determine contextual information in a microscope image through classification. For example, different recording methods or different sample types can be used during model training, and the model will classify them according to these methods or sample types. The text-to-image model computes an image consistent with the contextual information (=reference image), for example, an image with the same type of recording and the same sample type as the image shown in the microscope image. However, the reference image is not necessarily registered to the microscope image. However, the reference image can be used to determine the brightness value transformation of the microscope image.

[0042] An image generator can contain an encoder that computes an embedding or feature vector from the input microscope image, such as using an (auto)encoder, a ViT encoder, or PCA (principal component analysis). The image generator also contains a decoder that computes an output image (=reference image) based on the embedded image. Alternatively, an LLM can be used instead of an encoder to generate an image description as input to the image generator, which is then designed as a text-to-image model. In particular, the image generator can be a generative and / or diffusion model of a GAN.

[0043] Obtain reference images through image retrieval. Determining at least one reference image for a microscope image may require querying a reference image from an image database based on the microscope image and / or contextual information.

[0044] The image database contains several images with the desired image attributes or brightness value attributes. At least one image with the greatest similarity or correspondence to the microscope image is selected from the database. Because the image database only contains images with the desired image attributes, the selected image is suitable as a reference image for determining the brightness value transformation of the microscope image. The image selected from the database does not need to be registered with the microscope image.

[0045] Database queries can be implemented as image retrieval using embedding models, specifically machine learning models that compute embedding elements of microscope images in a low-dimensional embedding space (with dimensions lower than the microscope image's dimension). Each image in the database has one embedding element stored in the same embedding space. For each microscope image, at least one image is selected where the distance between the database embedding element and the microscope image embedding element is minimized. This distance is measured in the embedding space. This ensures that at least one image similar to the microscope image can be selected. Alternatively, instead of embedding the image itself, contextual information about the image can be embedded. For example, contextual information about the image can be generated from the corresponding image using an LLM or classification model, as described in detail earlier. The database can be subsequently expanded without retraining the embedding model or any other model.

[0046] To simplify the language, various embodiments are described with reference images. More generally, one or more reference images may be used. For example, brightness target data may be obtained from each reference image according to the described method, and then averaged.

[0047] Context information The contextual information of a microscope image can specify one or more of the following: the microscopy method or contrast type used when the microscope image was taken; the sample type, sample conditions, or sample processing of the sample mapped in the microscope image.

[0048] Different microscopy methods include, for example, optical microscopy, electron microscopy, X-ray microscopy, scanning probe microscopy, ion microscopy, or ultrasound microscopy. For example, different types of contrast can include bright-field, fluorescence, or phase contrast (which may be further subdivided into different phase contrast methods, such as Zernike phase contrast, DIC, or Hoffman modulation contrast). Sample type can specify the type of sample being examined, such as distinguishing different types of rocks, cell types in biological cells, or electronic components being examined. For example, sample preparation methods can specify the dyes and / or other preparation steps used. Sample condition can indicate the age, bleaching status, or health condition of biological cells, or cell state (e.g., mitosis, apoptosis).

[0049] Contextual information can specify the type of artifact, either alone or additionally, and can also selectively specify the location of the artifact (e.g., hammering). Contextual information can also include the type of experiment in which the microscope images were taken, or the type of biological application. Generally, contextual information defines which aspects are important when displaying image data.

[0050] Contextual information can also indicate the location of important image regions, for example, regions that should be well controlled. For instance, a segmentation model can be used to determine image regions, which calculates the association between each image pixel and a specific object type. In principle, contextual information can also include the microscope image itself, or image data derived from it, such as a low-resolution version of the image. User information (such as user ID or history) can also be used as additional contextual information.

[0051] Contextual information can take many forms, particularly text, numerical data, image data, and / or feature vectors that abstractly describe the image content and can include all of the above, such as sample type and measurement method.

[0052] Determine context information For example, contextual information about a microscope image can be obtained by performing image analysis on the image and / or querying the metadata of the microscope image.

[0053] Metadata consists of textual or numerical information stored along with microscope images. For example, metadata can specify the microscope configuration or settings at the time the microscope image was taken, as well as dye or other sample properties.

[0054] Image analysis can be achieved, in particular, through machine learning models, or through classical image processing algorithms. The learning model can be based on deep learning, such as using CNNs or Transformer-based networks. A learning model can be designed to extract feature vectors that serve as a compressed representation of the microscope image, abstractly describing the image.

[0055] Furthermore, it can prompt microscope users to manually input contextual information, particularly through a language model that queries for missing contextual information based on existing contextual information. For example, if image analysis is sufficient to determine the type of the microscope image taken but not the sample type, the language model can ask the user for the sample type.

[0056] User input can take the form of free text, such as an experiment description. The LLM can selectively extract feature vectors from this description, which contain contextual information. Users can also be provided with various predefined settings to choose from, such as the option to choose between long-term or high-resolution time-lapse experiments, and / or to select the dye to use from a list of dyes. User choices will be used as contextual information.

[0057] In addition to microscope images, contextual information can be extracted from image data. For example, besides microscope images, at least one other image of the same sample can be taken, particularly an overview image or an image taken using different microscope settings or configurations. Such additional images can also provide contextual information about the microscope images; for example, the sample type or other sample characteristics can be determined from the overview image.

[0058] Contextual information can typically be determined using various machine learning models, such as classification, segmentation, detection, regression, or anomaly detection / single-class classification models. These models can be designed to process image data; in particular, classification, regression, or anomaly detection models can also be configured to process other types of data.

[0059] Brightness value conversion A brightness value transformation can be a mathematical mapping from a given brightness value to a new brightness value. Therefore, each (original) brightness value is assigned a new brightness value, which can typically be the same as or different from the original. This mapping can be a monotonic function. This means that the brighter a pixel in a microscope image, the brighter the transformed pixel will be, although multiple different brightness values ​​can also map to the same new brightness value. This function can be applied to all pixels (image points) of a microscope image, thus mapping each pixel's brightness value to a new brightness value using the same rules.

[0060] Brightness value transformation can be achieved based on power functions (gamma correction) or other functions (especially monotonic functions).

[0061] For example, the mathematical function or transformation model used for luminance value transformation may have the following parameters whose values ​​need to be determined: • Two values, specifying the lower and upper percentiles of the brightness value, respectively, to normalize the brightness value to its minimum / maximum value, i.e., the offset of the black and white points; and / or • Minimum value (all brightness values ​​below this value are set to black), maximum value (all brightness values ​​above this value are set to white), and gamma value / gamma correction (where brightness values ​​are adjusted using a power function, and the gamma value specifies the power; depending on the gamma value, brightening or darkening occurs at darker or brighter brightness values, as well as selective contrast enhancement); and / or • Define a fixed or variable number of control points for grayscale transformation: The function / display curve assigns a new brightness value to each brightness value, where the support points are set at specific brightness values ​​and assigned a new brightness value; otherwise, a continuous or smooth function is applied through the support points. • Histogram balancing parameters; in the case of CLAHE, these parameters can be clipping limits and / or tile size.

[0062] The parameters of the brightness value transformation can be determined analytically or iteratively, for example, through gradient descent and error backpropagation. The parameters can also be determined by direct estimation, for example, using classic machine learning models and / or deep neural networks, such as CNNs or Transformer-based models. Furthermore, the parameters can be determined through regression or ordinal regression. The parameters of the brightness value transformation can also be determined using a model trained through reinforcement learning.

[0063] For all machine learning models mentioned as examples of defined parameters, it is also possible to train the model using only high-quality images, such as images obtained from technical articles or microscope advertising materials.

[0064] The parameter values ​​for brightness value transformation can be set, in particular, through contextual information and / or the brightness value frequency distribution of the microscope image or the microscope image (without requiring the actual image data of the microscope image). Image data or image content can also be encoded and used through feature vectors.

[0065] A luminance value transformation increases the contrast of at least one original luminance value range (e.g., dark pixels) while decreasing the contrast of at least one other luminance value range (e.g., bright pixels). Based on the microscopic image and / or contextual information, it can be determined in which luminance value transformations should increase contrast and in which luminance value ranges should decrease contrast.

[0066] Brightness value adjustments can also be achieved through pre-selection / preset display parameters. Display parameters can include gamma value, white point, and black point, etc. These parameters can be adjusted by the user, and details will be provided later.

[0067] The brightness value transformation is not a threshold comparison like some classic segmentation algorithms, but rather maps to a series of values ​​(e.g., at least 16 or 256 distinct values). To selectively enhance contrast, the slope of the image is progressively greater than 1 (when the image is rendered between two scales of the same color depth, e.g., from an initial 256 distinct pixel values ​​to still 256 distinct pixel values).

[0068] Brightness value transformation using a registered reference image For example, when using a reference image registered with a microscope image, the parameter values ​​for the brightness value transformation are determined. This might happen, for example, if the reference image was derived from the microscope image using the image-by-image model described above. The parameter values ​​that need to be determined could be gamma values, black points, and white points. The parameter values ​​are determined through optimization: determining the pixel differences between the microscope image and the reference image after applying the brightness value transformation (requiring optimization of initial parameter values). For pixel differences, the differences in the matching image coordinates in the two images are recorded, and these differences for all pixels are summed. Pixel loss functions (loss functions) can be used to capture these pixel differences, such as the L1 norm (sum of absolute differences in pixel values), the L2 norm (sum of squared differences in pixel values), or SSIM (Structural Similarity Index). The parameter values ​​are optimized to minimize the differences from the reference image. Therefore, the overall impression of the reference image can be mimicked, but the fine details (artifacts / illusions) of the reference image cannot be reproduced because the microscope image can only be modified using a few parameters of the brightness value transformation.

[0069] Brightness value transformation using an unregistered reference image In some embodiments, a reference image is used, but this reference image is not registered with the microscope image. This may occur, for example, if the reference image was obtained from a database using image retrieval, or if the reference image was generated from a text image description using a generative model. The parameter values ​​for the brightness value transformation are determined iteratively, similar to the previous example. To do this, the microscope image with the brightness value transformation and preset parameter values ​​is compared with the reference image. Since there is no registration between the images, pixel-by-pixel comparison is not possible. Instead, optimization can be performed using a "style loss," where the loss function compares the image impression rather than the presented structure. The parameter values ​​are adjusted until the style loss is minimized. Alternatively, in addition to the style loss, a discriminator can be used, like in a GAN. The discriminator determines whether the microscope image with the brightness value transformation using the preset parameter values ​​is in the same domain as the reference image. It functions similarly to the discriminator in CycleGAN. In principle, the discriminator can be trained on the image in real time or pre-trained. The parameter values ​​for the brightness value transformation are adjusted so that the discriminator cannot reliably detect differences in image type (domain) with the reference image. Alternatively or additionally, a loss function based on an artifact detection model can be used, which indicates the location of image artifacts (e.g., hammer marks) in the image. The parameter values ​​of the brightness value transformation are adjusted to minimize the loss. In this way, the brightness value transformation can suppress artifacts contained in microscope images, i.e., reduce the contrast of these structures, or by altering black / white point portions or completely eliminating image information related to artifacts. For example, hammering artifacts can be suppressed by increasing black points to set dark pixels to black. Therefore, the procedure can automatically determine appropriate parameter values ​​to suppress or hide these unwanted image contents when hammer marks (automatically detected) are present. At the same time, the brightness value transformation ensures that no new image content is generated.

[0070] Images displayed during microscope use Various traditional methods for improving image quality are presented using machine learning models, simultaneously displaying the original microscope image and the processed image. This comparison allows users to assess whether the image processing was performed correctly, particularly whether any serious artifacts were produced.

[0071] In contrast, brightness value transformation is suitable for directly enhancing and displaying recorded microscope images. Since brightness value transformation does not add any new, erroneous image content, there is no need to provide the user with a comparison to the original image.

[0072] In various variations of this invention, the microscope images are captured using a microscope, and the display of the microscope images only occurs after a brightness value transformation is applied. This method is also applicable to real-time images, i.e., when a camera continuously captures images and displays these images on a screen as video. In particular, the microscope images are displayed on the screen as real-time images immediately after recording and brightness value transformation.

[0073] To display the image on the screen, it can also automatically determine which display color the image or its corresponding channel should have. The display color is derived from contextual information. For example, different display colors can be defined for different dyes, fluorescence wavelengths, and / or sample types. The value calculated through brightness value conversion is related to the brightness of the selected display color. For example, it can be specified that a blue hue should be selected for an image containing DAPI dye.

[0074] In addition to the microscope image after brightness value transformation, the corresponding brightness value histogram and / or histogram of the original image can be displayed, along with visualized transformation parameters (e.g., curves). Individual transformation parameters can also be displayed.

[0075] In addition, confidence levels or confidence plots can be calculated and displayed. Confidence plots show the reliability or certainty of the optimized image. If the confidence level is too low, the image may not be displayed at all, or the original microscope image may be displayed, or a minimum / maximum normalized image or a percentile normalized image may be displayed.

[0076] For example, confidence can be determined based on the confidence probability value predicted by the model (if the parameters are directly predicted), or based on the difference between previous and optimized images, and / or by an evaluation model that receives optimized microscope images as input and is trained to determine the confidence level.

[0077] In various improvements related to image display according to variations of the present invention, it can also be configured such that, in addition to displaying a microscope image optimized by brightness value transformation, or as a supplement to displaying a microscope image optimized by brightness value transformation, the microscope image optimized by brightness value transformation is also passed to a subsequent image processing program. For example, an image processing program can be designed to segment or count specific objects. In this case, if the brightness value transformation can improve the visibility / contrast of the object and reduce the visibility of the background structure, the accuracy of the image processing program can be improved.

[0078] Feedback regarding modifications to context information or brightness value transformations Optionally, the contextual information used to determine the brightness value transformation, along with the microscope image to which the brightness value transformation was applied, is also displayed to the user. The user can input options to change the contextual information used. Input options can be implemented via dialog boxes, list selection, or free text input, or via voice input and / or with the aid of LLM. For example, the detected sample type can be specified as contextual information, and the user can select or input different sample types. This is particularly useful if the detected sample contains different cell types, but only one is automatically detected. If the contextual information changes, the brightness value transformation is recalculated (using the changed contextual information) and applied to the microscope image. The optimization of the microscope image update through brightness value transformation is essentially performed in real time.

[0079] More generally, users can provide additional / contextual information as feedback on the microscope image optimized through brightness value transformation, supplementing or replacing the contextual information. As mentioned above, feedback can be provided via dialog boxes or communication with the LLM, where the LLM evaluates the user input. For example, a user can indicate that the brightness or contrast of certain objects is too high / too low, or that certain objects should stand out more from the background. Based on this feedback, the parameter values ​​of the brightness value transformation will be adjusted accordingly, for example, increasing the contrast of the user-specified object and increasing the brightness difference between the object and the background.

[0080] A range of possible feedback options can be provided for the user to choose from. The list can include options such as: "Overall image too bright"; "Sample area too bright"; "Overall image contrast too low"; "Sample area contrast too low", etc.

[0081] For example, a possible free text input for LLM analysis could be: "Hammering artifacts are still visible. Please adjust the settings to make them invisible." If the user points out that the artifacts are too distracting, the parameters of the luminance value transformation would be adjusted to reduce the visibility of the artifacts (specifically by reducing the contrast of the luminance value range primarily used by the artifacts, and / or bringing the luminance value range primarily used by the artifacts closer to the background). To do this, for example, detection / segmentation could be performed for the user-specified artifact type (e.g., hammering artifacts); for luminance values ​​that mainly appear in this identified image area, the contrast should be reduced; and the display curve should be adjusted accordingly to make the slope of that luminance value range gentler, i.e., mapping the luminance value range to a new, smaller luminance value range.

[0082] Users can also provide feedback by selecting (e.g., clicking) a specific object in a microscope image optimized for brightness value transformation. A user can select an object and specify how its visibility should be changed, specifically increasing or decreasing its visibility. In DAPI fluorescence images, surrounding cellular structures are visible in addition to the cell nucleus; a user can click on these structures and specify that they should be suppressed more strongly. Image analysis (e.g., segmentation) can be used to determine the image region displaying the type of selected object (e.g., cellular structures outside the cell nucleus). The parameters of the brightness value transformation can be adjusted to reduce contrast for brightness value ranges that occur more frequently in the cellular structure image region than in the cell nucleus image region. Specific objects can also be selected via eye tracking, i.e., by recording the user's eyes through a camera to determine which object in the microscope image the user is looking at. This allows for interactive transformation and display of the brightness values ​​in the microscope image.

[0083] User feedback can also directly influence the parameters of the brightness value transformation, thus changing the parameter values ​​accordingly. To help address this issue, a display curve can be shown, which graphically indicates the function that maps the brightness values ​​of the original microscope image to the new brightness values.

[0084] Feedback or contextual information can be used when a user (especially for newly created samples) manually adjusts the calculated display curve or manually sets the display curve for a microscope image for the first time. Subsequent experiments will automatically mimic the image changes caused by this display curve. Since the final image changes largely depend on the histogram of the corresponding microscope image, the same display curve will not always be used. For each new microscope image, the display curve / brightness value transformation is newly optimized while keeping the target attributes unchanged. For example, if the user maximizes the contrast of the cell nucleus while reducing the contrast of other image content, the brightness value transformation of subsequent microscope images will also produce this effect.

[0085] Machine learning models The machine learning models described in this article can learn from training data and can be, or include, neural networks, particularly deep neural networks. The models can be trained using supervised learning, unsupervised learning, or semi-supervised learning methods. Reinforcement learning is also feasible.

[0086] In various training variations, the objective function is optimized; for example, the loss function is minimized or the reward function is maximized. For ease of understanding, some embodiments use a loss function, but these embodiments can also be modified by optimizing different objective functions (e.g., reward functions). Iterative tuning of model parameters can be accomplished using optimizers, such as gradient-based methods (e.g., Adam) or other gradient descent methods (e.g., SGD (Stochastic Gradient Descent)). Backpropagation methods can adjust parameter values ​​based on the current result of the objective function. Embodiments described using gradient descent can be improved by using simulated annealing as an alternative or complementary method.

[0087] A model can optionally contain multiple individual models / sub-models, which can be trained individually or together. Furthermore, models can be designed as ensemble models. A model can also be understood as several independently trained models whose results are combined through majority voting.

[0088] Furthermore, model distillation can be used, where a teacher model is learned and a student model is learned through its soft output. Therefore, the model described herein can be learned as a student model in the process of improvement.

[0089] The model can incorporate CNNs, and in particular, it can be learned within a GAN (Generative Adversarial Network) architecture, especially StyleGAN or Alias-free GAN. Diffusion models, autoencoders (e.g., variational autoencoders / VAEs or Transformer-based autoencoders) or Transformer-based encoders, decoders, or encoder-decoders are also feasible.

[0090] In training a diffusion model (denoising diffusion model), different levels of noise are added to the training images. The model learns how to denoise or separate the image into noise and signal components, thereby learning image synthesis. The feature space of a diffusion model can have semantic meaning. Diffusion models can incorporate CNNs, or use other processing layers instead of CNNs, particularly MLP mixers with multilayer perceptrons or attention-based modules, such as the Transformer in ImageBART.

[0091] The described model can be modified to be designed for RAG (Retrieval Augmentation Generative Architecture), enabling the model to query desired information from external databases. RAG visualization (multimodal) can also be performed using image data from the database. This allows for easy addition of data to the database and its integration with the model after training, without requiring new model training.

[0092] General characteristics Microscope images can be understood as images taken with a microscope or images calculated using microscope measurement data. Microscope images can be formed, in particular, from one or more original images or further processed images from a microscope, and can include two-dimensional image data, three-dimensional image stacks or volumetric data, or time-series data, where two-dimensional or three-dimensional image data of the same object are recorded at different times. Microscope images can be obtained through a microscope software interface or storage medium. Acquisition can be triggered manually by the user or automatically, for example, as part of a workflow.

[0093] The structures or objects depicted in a microscope image can be arbitrary in principle. In addition to the sample itself (such as biological structures, electronic components, or rock parts), sample containers, sample carriers, and microscope components (such as the sample stage or areas thereof) can also be shown.

[0094] A microscope can be an optical microscope equipped with a system camera, or optionally with a separate overview camera. The overview camera or its objectives may be non-telecentric. Other types of microscopes include electron microscopes, X-ray microscopes, or atomic force microscopes. A microscope system is a device that includes at least a computing unit and a microscope.

[0095] The computing device can be designed to be distributed, serving as a physical component of the microscope, located independently within the microscope environment, or situated at an arbitrary distance from the microscope. Generally, it can consist of any combination of electronic components and software, and in particular can include computers, servers, cloud computing systems, or one or more microprocessors or graphics processors. The computing device can also be configured to control microscope components.

[0096] In this context, a label is understood as a predefined result, as it is computed by the model being trained based on inputs belonging to that label. During training, the difference between the model output and the labels is used to adjust the model parameters / weights. Depending on the model, labels can be images containing image data or segmentation masks, or they can consist of other data, such as numerical values ​​or classifications by object type or size. In contrast to labels, contextual statements refer to the data fed into the model as part of the input data so that the model output can be computed based on that data.

[0097] Phrases such as “based on,” “dependent on,” or “depends on” should not be interpreted as exhaustive, as further dependencies may exist. Furthermore, intermediate steps may be included: for example, determining the luminance value transformation based on contextual information can be achieved in two ways: directly using the contextual information to define the parameter values ​​of the luminance value transformation; or using the contextual information to obtain other data (such as a reference image or reference histogram) and using only this data (not the contextual information itself) to define the parameter values ​​of the luminance value transformation.

[0098] The singular form of the description should cover both "exactly one" and "at least one". For example, if the use of a reference image is described, one or at least one reference image may be used.

[0099] The described method can be applied globally to the entire microscope image or locally to a specific portion of the image. Therefore, brightness value target data (especially image statistics) can be applied globally to the entire image or individually to a local image region. Similarly, the parameters of the transformation model / brightness value transformation can be applied globally to the entire image or to a local image region. Individual parameters of a local image region (e.g., divided into a grid) can be considered individually or determined smoothly / smoothly between adjacent regions. Thus, a spatially variable display curve applied to the image represents a kind of HDR (High Dynamic Range), through which contrast can be locally increased. If the same transformation is applied to the entire microscope image, the risk of distortion can be almost eliminated. If the brightness value transformation includes separate brightness value transformations for different image regions, the overall image impression may change significantly, but the display quality can be further improved. For example, some structures can be suppressed more effectively, while others can be displayed with higher contrast. For brightness value transformations of different image regions, each brightness value transformation can be determined in the manner described. The boundaries of different image regions can be defined according to the displayed content, for example, by one or more segmentation masks that differentiate based on the displayed sample / structure type or the artifacts contained. For example, image regions containing artifacts can be made to resemble the background more closely or have their contrast reduced, while image regions containing the studied sample can be displayed with higher contrast.

[0100] For ease of understanding, the embodiments use image channels (grayscale images) as an example. However, an image can contain any number of channels; for example, these channels may correspond to different fluorescence channels, or, in the case of a color image, to three colors (such as RGB). In these cases, the described procedure can be applied to any image channel. Therefore, a brightness value can be understood as the value of a specific image channel and does not necessarily represent the overall brightness produced by all image channels. In multi-channel images (e.g., color images), brightness value transformation can also be applied to brightness or other variables composed of multiple or all channels.

[0101] The characteristics of the additional device features described in this invention, when used as intended, will produce various variations of the method of this invention. Conversely, a microscope system, or in particular a computing device and / or computer program, can be configured to perform the method variations. The training process of the described machine learning model defines the properties of a fully trained model, wherein different method variations are defined by the fact that the described training process is a method step to be performed in the method according to the invention, or that a model trained in this manner is used for inference. Attached Figure Description

[0102] Other effects and features of the present invention will now be described with reference to the accompanying drawings: Figure 1 This illustration illustrates the process of traditional image correction. Figure 2 The process of an embodiment of the present invention is illustrated schematically; Figure 3 The process of another embodiment of the present invention is illustrated schematically; Figure 4 The process of another embodiment of the present invention is illustrated schematically; Figure 5 The process of another embodiment of the present invention is illustrated schematically; Figure 6 The process of another embodiment of the present invention is illustrated schematically; Figure 7 An embodiment of the microscope system according to the present invention is illustrated schematically. Detailed Implementation

[0103] Various embodiments are described below with reference to the accompanying drawings. Components that are identical or have similar functions are generally labeled with the same reference numerals.

[0104] Figure 1 Display calibration based on existing technology Figure 1 This illustration schematically shows how the display of a microscope image is traditionally manually calibrated.

[0105] In step S1, a microscope image 20 is acquired from microscope 1, and then displayed on screen B in step S2. Microscope image 20 is a fluorescence image showing two cell nuclei 21 and 22, and also showing extranuclear cellular structures 23 through crosstalk in the fluorescence channel.

[0106] Screen B also displays histogram H. Histogram H shows the frequency of occurrence of all possible brightness values ​​(pixel values) V1 in microscope image 20.

[0107] In step S3, the microscope user U manually sets the display curve D. Figure G1 shows the overlay of the histogram H and the display curve D. Figure G1 displays all possible brightness values ​​V1 (pixel values) of the original microscope image 20 on the horizontal axis. For example, the brightness value range can be from 0 to 255, with a color depth of 8 bits.

[0108] The display curve D, or transformation curve, is a function that assigns a new brightness value V2 to each brightness value V1 of the original microscope image 20. The vertical axis of Figure G1 shows the brightness value V2. If the display curve D is a straight line with a slope of 1 (as shown in Figure G1 for ease of understanding), the brightness values ​​of the microscope image 20 will not change due to the display curve D.

[0109] Microscope user U can adjust the display curve D to make the structure of interest more clearly visible. In this example, the experienced user U lowers the display curve D for dark pixel values, further deepening the dark pixels in microscope image 20. As a result, cell structure 23 appears darker. Furthermore, user U adjusts the white value W accordingly. The white value W specifies a brightness value above which all brightness values ​​in the original microscope image 20 should be rendered white. User U sets the white value W to the brightest brightness value appearing in microscope image 20.

[0110] In step S4, the display curve D is applied to the microscope image 20, that is, the brightness value V1 is mapped to the brightness value V2 according to the display curve D, thereby calculating the corrected microscope image 25. The corrected microscope image 25 is displayed on the screen B.

[0111] The procedures described require experience and expertise. Furthermore, even experts often cannot accurately apply manual adjustments to other images.

[0112] In traditional automated systems, the display curve is set based solely on simple image statistics, regardless of the structures displayed in the image or the image type. This typically results in a modest improvement in the contrast of all displayed structures, making the cell nuclei 21, 22, and the remaining cellular structures 23 more prominent relative to the background, but this is not consistent with the goals of the current application.

[0113] In contrast, the present invention is able to produce higher quality images, as detailed below.

[0114] Figure 2 Brightness value transformation by registering a reference image. Figure 2 An embodiment of a computer-implemented method for improving the quality of microscope images according to the present invention is illustrated schematically.

[0115] In process P1, a microscope image 20 is acquired, for example, an image taken by microscope 1. Microscope image 20 can be the original image or an image that has been processed from the original data. In this example, it is a fluorescence image in which two cell nuclei 21 and 22 are visualized using DAPI dye. Undesirablely, other cellular structures 23 may also be visible outside the cell nuclei.

[0116] In process P2, at least one reference image 20R is determined for the microscope image 20. To do this, the microscope image 20 from process P2a is input into the image-to-image model M1, which is trained to compute a quality-improved output image from the input image, i.e., process P2b. The output image is then referred to as the reference image 20R. For example, the image-to-image model M1 can be trained under supervised supervision, where the input images used in training have the same domain (image type and sample type) as the microscope image. Compared to the input image, the quality of the predefined target image / annotation used in training is improved; for example, the visibility / contrast of structures of interest is enhanced, background structures are hidden, and recording artifacts are reduced.

[0117] In the example shown, the image-to-image model M1 calculates an output image (reference image 20R) from microscope image 20, in which cell structure 23 is less prominent, while the nuclei of interest 21 and 22 are displayed with higher contrast. However, an artifact A is incorrectly added; in this case, artifact A is a structure not included in microscope image 20. Furthermore, the internal structures of nuclei 21 and 22 in reference image 20R differ from the actual structures in microscope image 20.

[0118] Because the image quality is improved, the reference image 20R can still be used to optimize the microscope image 20. For this purpose, the brightness value transformation T is determined in process P6. In the example shown, the parameters of the brightness value transformation T are iteratively adjusted. The brightness value transformation T maps the brightness values ​​of the microscope image 20 to new brightness values. Each brightness value is assigned a new brightness value (which may be different from or the same as the original brightness value). The image is described by one or more parameters, and the values ​​of these parameters need to be determined. For ease of illustration, the brightness value transformation T is shown in the middle of the figure: in this example, it can be represented by a function (transformation characteristic / display curve D) that assigns a new brightness value V2 to each brightness value V1 of the microscope image 20. The brightness value transformation T, or display curve D, is defined by multiple adjustable parameters P, which in this example include the black point S, the white point W, and the gamma value G, or it could be at least one reference point on the display curve D. As mentioned earlier, the black point S and the white point W specify the thresholds, where all pixel values ​​below the black point S are mapped to black (i.e., brightness value V2 = 0), and all pixel values ​​above the white point W are mapped to white (i.e., brightness value V2 = maximum value). The gamma value G can represent the power of the brightness value V1 mapped to a new brightness value, where the brightness value V1 can be converted to a scale from zero to one. The V2 value of the display curve D is 0 for all V1 values. When V1 > W, V2 remains unchanged, V2 = maximum value. The slope of the display curve D is not linear, but is determined by a predefined function, such as a power function like V1^Gamma.

[0119] In principle, the black point S can also be set to a value of V2>0 (i.e., V2=ps, where ps>0), and / or the white point W can also be set to a value of V2<maximum value (i.e., V2=pw, where pw<maximum value), so that the resulting image only uses a limited range of brightness values ​​from ps to pw.

[0120] To iteratively adjust the parameters P (e.g., black point S, white point W, and gamma value G) of the brightness value transformation T, the brightness value transformation T is first applied to the microscope image 20 with initial parameter values. The initial parameter values ​​can be fixed (e.g., black point S = 0, white point W = maximum value, gamma value G = 1 or a number other than 1), or they can be set according to the microscope image 20 (e.g., the initial value of black point S = the minimum pixel value appearing in the microscope image 20; the initial value of white point W = the maximum pixel value appearing in the microscope image 20).

[0121] In process P7, a brightness value transformation T (initially using the initial values) is applied to microscope image 20, resulting in an image 26 with modified brightness values. A loss function L captures the difference between image 26 and the reference image 20R. In this example, the difference calculated by loss function L consists of pixel-level differences between image 26 and the reference image 20R. The result of loss function L is used by optimizer O to adjust the value of parameter P of brightness value transformation T. For example, gradient descent can be used to determine whether the current parameter value should be increased or decreased to reduce the difference detected by loss function L. Optimizer O can also specify / influence the step size, adjusting the parameter values ​​in each iteration. After changing the parameter values, in process P7, a new image 26 is calculated again by applying brightness value transformation T to microscope image 20.

[0122] Iterative adjustment is similar to a typical learning process in a neural network, but in this example, the brightness value transformation T represents only one layer, so there is no need to backpropagate through multiple layers to adjust the parameter values.

[0123] When the stopping criterion is met, the adjustment of the iteration parameter values ​​ends. For example, when a certain number of iterations are reached and / or the reduction in the difference captured by the loss function L in multiple iterations is less than a certain threshold.

[0124] The value of parameter P determined in this way is now used for the luminance value transformation T. In process P10, the luminance value transformation T is applied to the microscope image 20 to obtain the corrected image 27, i.e., process P11. By changing the luminance values, the corrected image 27 is adjusted to match the microscope image 20. However, no new structure / image content can be added through the luminance value transformation T. This eliminates the risk of hallucinations and additional artifacts A, as seen in image learning to image model M1. However, as mentioned earlier, the reference image 20R with artifacts A can be used to determine suitable parameter values ​​for the luminance value transformation T.

[0125] The corrected image 27 is displayed on screen B in process P12. Alternatively, this process can be performed immediately after image acquisition in process P1. This allows for real-time display of the corrected image 27, minimizing the time delay compared to data acquisition.

[0126] The brightness value transformation T can also be represented on screen B, for example, as shown in curve D, which overlays a histogram of brightness values ​​from microscope image 20. The microscope user can then select and adjust specific parameter values ​​for the brightness value transformation T.

[0127] The described procedure differs from the simple automatic adjustment of the parameter values ​​of the display curve D not only in the various processes but also in the results obtained. For example, a simple automatic function can optimize the contrast of all display structures. In contrast, the display curve D shown here reduces the contrast of dark structures or completely hides them. This suppresses unwanted cellular structures 23; however, through simple automatic processing, these cellular structures are more clearly visible in the generated image.

[0128] In a variation of the illustrated embodiment, more than one reference image 20R is used. For example, variable noise can be added to the microscope image 20, or the microscope image 20 can be slightly modified in other ways, so that the image-to-image model M1 calculates different reference images 20R from the same microscope image 20. Alternatively, multiple slightly different image-to-image models or a single model from a set of models can be used to calculate multiple slightly different reference images 20R from the same microscope image 20. In the adjustment process P6, different reference images 20R can be used iteratively for adjustment. Alternatively, the average value of the reference images 20R can be used in process P6.

[0129] In a further improvement, process P6 involves setting / analyzing and determining parameter values ​​in a single step, rather than iteratively adjusting them. This is because the brightness value transformation T can be achieved using a simple surjective function.

[0130] In a further improvement, in the (first) use of the loss function L, the microscope image 20 is used instead of the modified image 26, which corresponds to the brightness transformation T, where the display curve is a straight line with a slope of 1.

[0131] Figure 3 Brightness value transformation using an unregistered reference image Figure 3 Another embodiment of the method according to the present invention is illustrated schematically.

[0132] Figure 3 Zhongyu Figure 2 Elements with the same reference numerals in the figures can be arranged according to... Figure 2 The design is based on the method described above, and the following is only a brief explanation.

[0133] exist Figure 3 In this process, a microscope image 20 was acquired, and a reference image 20R was determined based on it. However, in this example, the reference image 20R was not registered with the microscope image 20. This means that pixels with the same image coordinates in the microscope image 20 and the reference image 20R do not necessarily show the same structure.

[0134] Microscope image 20 is input into a language model (large language model) LLM. In process P3, the LLM generates a description of microscope image 20, i.e., determines the contextual information K of microscope image 20. The description can be in text form and may include the type of image captured (fluorescent image), characteristics associated with the captured image (DAPI dye), and / or the structures displayed (two cell nuclei). In process P8, the contextual information K is input into an image generator Gen (here, a text-to-image model), which generates a reference image 20R based on the contextual information K in process P9. Processes P3, P4, P8, and P9 together constitute process P2, used to determine at least one reference image 20R for microscope image 20.

[0135] Reference image 20R is consistent with the contextual information K; therefore, in this example, it is a fluorescence image of two cell nuclei stained with DAPI. However, the size, shape, and location of these cell nuclei differ from those of cell nuclei 21 and 22 in microscope image 20. Nevertheless, reference image 20R can be used to determine a suitable value for the parameter P of the brightness value transformation T.

[0136] Image 26 is obtained by performing a brightness value transformation T on microscope image 20 using initial parameter values, and this image is then input into loss function L. Additionally, reference image 20R is input into loss function L. Loss function L does not calculate pixel-level loss because the two images are not registered. Instead, loss function L calculates style loss. Style loss detects differences in display manner. However, the location of the depicted structure is not increased, or almost not increased, in fact, the differences reflected by style loss are minimal. For example, style loss can be calculated by comparing the histograms of the two images. Alternatively, style loss can be determined by comparing the texture of the images: calculating texture features of the two images, such as contrast, uniformity, and entropy, for example using a gray-level co-occurrence matrix (GLCM). The gray-level co-occurrence matrix (GLCM) represents the frequency of certain gray-level value pairs in an image. The loss function can quantify the differences between texture-based features extracted from the gray-level co-occurrence matrix. Alternatively or additionally, edge detection can also be performed, and style loss can compare the number of edges or other edge properties of the two images. In this way, loss function L can detect style differences between two unregistered images. Based on the result of the loss function L, optimizer O changes the value of parameter P of the brightness value transformation T to minimize the loss function L. The value of parameter P is determined through iterative adjustment.

[0137] Subsequently, in processes P10 and P11, the corrected image 27 can be calculated from the microscope image 20 using the luminance value transformation T, and displayed on screen B in process P12. Optionally, the context information K used in the luminance value transformation T is also displayed on screen B. User U can choose to provide feedback (feedback FB) to the corrected image 27 or the context information K used. In the example shown, the input option is implemented by communicating with a language model LLM, which calculates new context information K2 based on user U's feedback FB. The new context information K2 is then used together with the original context information K in process P8, and the process is executed again to generate a new corrected image 27 that takes into account the context information K2.

[0138] User U can provide feedback FB, for example, indicating that the displayed object (here, the cell nucleus) is overexposed, or that the context information indicates the object type is incorrect. Using this feedback FB, image generator Gen can create a new reference image 20R containing the underexposed object or the corrected object type.

[0139] In addition to, or as a supplement to, LLM, feedback input options can be implemented by allowing users to select points, regions, or objects in the corrected image 27, along with display requirements (e.g., "too bright," "increase contrast," or "suppress structure"). The feedback information FB contains location information and instructions on how to modify the image data. The location information can be selectively used to determine object / structure types, for example, by segmentation or other methods to determine which types of objects exist in the image region. As new contextual information K2, a new reference image can be determined by combining the determined object / structure type with the representation specification, rather than the location information.

[0140] If there is a conflict between the new context information K2 and the original context information K, the new context information K2 can replace the original context information K (or a portion thereof). User feedback FB can optionally be used directly as the new context information K2. The steps described for using feedback FB can also be applied to all other implementation variations. In this case, the new context information K2 will not be used by the image generator, but can be used as input to another learned model, or for database queries, or after embedding into the feature space, depending on the implementation.

[0141] In a variation of the illustrated embodiment, the context information K is not directly input into the image generator Gen. Instead, the encoder (e.g., a Transformer encoder) creates a mapping of the context information K to the feature space and inputs this mapping into the image generator Gen.

[0142] Figure 4Reference images are from a database; microscope images are embedded. Improvements to the foregoing embodiments Figure 4 describe.

[0143] In this scenario, LLM does not generate a text description of the microscope image 20. Instead, in process P4, an encoder Enc (e.g., a Transformer encoder) is used to compute a feature vector F from the microscope image 20. The feature vector F represents a compact, abstract numerical description of the microscope image 20. Optionally, the encoder Enc can also be designed as a multimodal encoder, receiving text data in addition to image data. For example, the text data could be metadata of the microscope image 20, containing contextual information.

[0144] Unlike the image generator in the previous illustrated example, Figure 4 An image database DB is used to store the images. The image in the database DB that is closest to microscope image 20 should be selected as the reference image 20R, procedure P9'. For this selection, each image in the image database DB stores a corresponding feature vector in the database DB, where the corresponding feature vector can also be calculated using the encoder Enc. At least the image in the image database DB with the smallest distance between its feature vector and feature vector F should be selected as the reference image 20R.

[0145] The style loss described in the figure above can be used to compare the reference image 20R and the microscope image 20. However, in the example shown, a discriminator Dis is used. The discriminator Dis receives the reference image 20R and the microscope image 20 as input and evaluates whether the two images come from the same domain, or whether they are similar in terms of image type or image attributes and sample type. Similar to the style loss in the figure above, based on the results of the discriminator Dis, the value of parameter P of the brightness value transformation T is adjusted by the optimizer O to minimize the differences detected by the discriminator Dis. The discriminator Dis can be a neural network, particularly including a CNN and / or a Transformer encoder. For example, the discriminator can be trained in a GAN setting.

[0146] In principle, a machine learning anomaly detector can be used instead of a discriminator, which receives the microscope image 20 as input. From a number of possible anomaly detectors, an anomaly detector can be selected based on a chosen reference image 20R, so that the selected anomaly detector can detect deviations of the type from the reference image 20R.

[0147] A combination of the described graphical examples can be used. For example, Figure 3The medium-style loss can be replaced by a discriminator, and vice versa. The image generator Gen can be replaced by the image database BD to select a suitable image from the image database BD based on the context information K. In this case, the image database contains context information for each stored image.

[0148] The embodiment in the illustrated example is further improved by not determining the deviation between the microscope image 20 and the reference image 20R, but instead recording the brightness value data of the microscope image and the target brightness value data (i.e., the brightness value data of the reference image), for example, through a loss function L. The brightness value data can indicate the uniformity, entropy, and / or dispersion of the frequency distribution of the corresponding image brightness values. The parameter values ​​of the brightness value transformation can be set iteratively or analytically to reduce any deviation between the above properties.

[0149] Figure 5 Brightness value conversion using a reference histogram Figure 5 Another embodiment of the method according to the present invention is illustrated schematically.

[0150] From microscope image 20, the language model LLM generates contextual information K in process P3. High-quality images and their respective contextual information are stored in database DB; alternatively, histograms of high-quality images and their respective contextual information are stored in database DB. An image / histogram that matches or is closest to the contextual information K of microscope image 20 is selected from database DB, process P13. The selected histogram is called the reference histogram RH. When an image is selected from database DB, its histogram will be used as the reference histogram RH.

[0151] In the reference histogram RH of the contextual information K (here: fluorescence images of the two cell nuclei), black or nearly black pixels appear significantly more frequently, while other brightness values ​​appear at roughly the same frequency, but significantly less frequently than black pixels. Therefore, most of the image is black or nearly black, while the cell nuclei utilize all brightness values ​​from dark gray to white, resulting in high contrast.

[0152] In contrast, the histogram H of the original microscope image 20 (as shown) Figure 5 The curve D in the figure shows a broad main peak at the dark gray pixel value, which is due to the visible cellular structure 23.

[0153] Microscope image 20 is adjusted using a brightness value transformation T to make the histogram of the resulting image 26 more similar to the reference histogram RH. As described in the previous figure, image 26 is calculated using the brightness value transformation T and initial or preliminary parameter values. The histogram 26H of image 26 should be as similar as possible to the reference histogram RH. The loss function L captures the difference between histogram 26H and the reference histogram RH.

[0154] The loss function L and optimizer O are used to change the value of parameter P of the brightness value transformation T to calculate a new image 26, which reduces the loss through its histogram 26H or iteratively minimizes the loss function L.

[0155] In the example shown, the brightness value transformation T increases the frequency of black or near-black pixels, while medium brightness values ​​(primarily corresponding to cell nuclei 21 and 22 in the microscope image) are stretched to a wider brightness range. This gradually brings the histogram H of the original microscope image closer to the reference histogram RH.

[0156] In this way, the parameter values ​​of the brightness value transformation T can also be determined, thereby allowing the calculation of the corrected image 27, as shown in the previous illustration.

[0157] In the above process, histogram 26H and reference histogram RH can be included in the loss function L as images or numbers (the brightness value frequency distribution 29 of microscope image 20 and the target brightness value frequency distribution 29R). Alternatively, one or more histogram attributes of histogram 26H (or frequency distribution attributes of microscope image 20) and histogram attributes of reference histogram RH (or target frequency distribution attributes) can be included in the loss function L. For example, these attributes can specify the dispersion, uniformity, entropy, and / or brightness mean of the corresponding frequency distribution / histogram. The attributes of the reference histogram RH input into the loss function L are also referred to as the brightness value target data 28R. Similarly, the attributes of histogram H included in the loss function L are referred to as brightness value data 28. Therefore, the loss function L can capture only attributes such as diffusion and brightness mean, rather than the entire histogram, which also makes the desired approximation possible. The described process P13 can be understood more generally as determining the brightness value target data 28R based on context information K, rather than determining the reference histogram RH from the database DB. Database DB can contain target data for other brightness values, rather than reference images or frequency distributions / histograms.

[0158] Based on the context information K, it can be determined which image attribute or type of luminance value target data should be optimized. Similarly, based on the context information K, the procedure can be used to determine which parameters of the luminance value transformation should be adjusted. If the context information indicates the presence of artifacts or other unwanted image content in the image, and the adjustable parameters contain black and / or white points, then suppressing these contents may be practical. In other cases, other adjustable parameters may be more advantageous, for example, determining the number of support points for the display curve based on the context information.

[0159] Figure 6 A database containing brightness value conversion parameters. Figure 6 Another embodiment of the method according to the present invention is illustrated schematically.

[0160] In this example, machine learning model M2 analyzes microscope image 20 to determine relevant contextual information K in process P3. For example, model M2 could be a classification model designed to classify based on different sample types and / or different contrast types.

[0161] The context information K is used for database queries in procedure P8'. In this example, the database DB stores the set of parameter values ​​for the brightness value transformation T, along with various context information. In procedure P13, a set of parameter values ​​is selected from the database DB whose associated context information best matches the context information K of the microscope image 20.

[0162] A database (DB) can store content in a tabular format. To determine matches or similarities between contextual information, this information can be selectively mapped to a feature space / embedding space using an embedding model (specifically, a Transformer encoder in machine learning). In this space, the distance between two embedding elements measures the difference in relevant contextual information. A database (DB) can contain embedding elements so that these elements do not need to be recalculated for each database query.

[0163] Optionally, it can also be specified that the histogram H or related histogram attributes are also used for database queries in process P8'. This allows the use of properties of the brightness value distribution to query suitable parameter values ​​for the brightness value transformation T, which might not be captured by the context information K. In this embodiment, the database DB also contains histograms or related histogram attributes. This selects from the database those parameter values ​​whose corresponding histograms / histogram attributes and the context information stored in the database DB are most similar to the context information K and histogram H of the microscope image 20.

[0164] In the illustrated embodiment, the database DB contains target brightness value data, rather than parameter values. Database filtering is performed as described above. The selected target brightness value data is then approximated to the brightness value data of the microscope image to determine the parameter values ​​of the brightness value transformation T.

[0165] These procedures can also be used to determine a luminance value transformation T with appropriate parameter values, through which an improved image 27 can be calculated and displayed.

[0166] Figure 7 :microscope Figure 7 An embodiment of a microscope system 100 according to the present invention is shown. It includes a microscope 1 and a computing device 17, which may be part of or separate from the microscope 1. The microscope 1 includes a tripod 2 on which other microscope components are fixed. It may include, in particular: an illumination device 3; a condenser 5 for directing illumination light to a sample area; an objective changer or turntable on which, in the illustrated example, an objective 6 is mounted; a sample stage 18 with a fixing frame 8 for securing a sample carrier 9; and a microscope camera 7. When the objective 6 is swung into the microscope beam path, the microscope camera 7 receives detection light from the sample area, where a sample 15 may be located, to capture a microscope image. The sample 15 may, in principle, be or include any object, fluid, or structure. To capture an overview image 14 of the sample environment as a microscope image, the microscope 1 may use a lower magnification objective and / or an optional additional overview camera (not shown), which can observe the sample area vertically or at an angle. To capture an overview image, an illumination system can be used, for example, a system comprising multiple LEDs 4 whose illumination direction is tilted and oriented relative to the detection axis of the dark-field measurement. In addition to the microscope camera 7, the sample can also be observed through the eyepiece 19. The microscope system 100 also includes a computer program 16, which is only schematically represented and stored on a non-volatile data storage device. The computer program 16, or computing device 17, is configured to perform variations of the methods described in other figures. In the example shown, microscope 1 is an optical microscope, but in principle it could also be other types of microscopes.

[0167] The various variations described with respect to the different accompanying drawings can be combined with each other. In all variations, processes can be added between the described or illustrated processes. Additional data can be added to the data used for calculations. For example, more contextual information can be added to the described contextual information, which can be obtained from image metadata, experimental descriptions, or by querying the configuration or parameters of the microscope system. The described embodiments are purely exemplary and can be modified within the scope of the appended claims.

[0168] List of reference numerals 1. Microscope 2. Tripod 3 lighting fixtures 4 LED 5 Concentrators 6. Objective lens 7. Microscope camera / system camera 8. Fixing frame for securing sample carriers 9 Sample carrier 14 Microscopic images / Overview images 15 samples 16 Computer Programs 17. Computing device 18 Sample Stages 19 Eyepieces 20 Microscopic images 20R Reference Image 21, 22 cell nuclei 23 Cell Structure 25 Corrected microscope images Image 26, calculated from the brightness values ​​of the current parameter values. 26H Histogram of Figure 26 27 Corrected images / microscope images 28. Brightness value data of microscope image 20 28R luminance value target data 29 Frequency distribution of brightness values ​​in microscope images 20 29R Target brightness value frequency distribution, such as the brightness value frequency distribution of a reference image. 100 Microscope Systems A. Image processing artifacts B screen D Display Curve DB Database / Image Database Discriminator Enc encoder F-microscope image feature vector Facebook user feedback Gamma value G1 diagram Gen Image Generator H histogram K Context Information K2 new context information, calculated based on feedback FB. L Loss function / Objective function LLM language model / Large language model M1 image-to-image model for calculating reference images from microscope images. The M2 model is used to calculate contextual information in microscope images. O optimizer P Parameters for brightness value transformation P1-P13 process RH Reference Histogram S Black Dot Steps of the traditional method (S1-S4) T brightness value conversion U microscope users Brightness value of microscope image 20 (V1) Brightness values ​​of 25 and 27 in the corrected microscope image (V2) W white spot

Claims

1. A computer-based method for improving the quality of microscope images, comprising: Acquire (P1) microscope image (20); Determine the contextual information (K) of the (P3) microscope image (20), wherein the contextual information (K) is at least related to the recording characteristics and / or the structure presented; Determine (P6) the luminance value transformation (T), which depends at least on the context information (K); and The brightness value transformation (T) is applied (P10) to the microscope image (20).

2. The method according to claim 1, in, Determining the luminance value transformation (T) includes: Determine the target luminance value (28R) based on the context information (K) (P13); Determine the brightness value transformation (T) so that the brightness value data (28) of the microscope image (20) approximates the brightness value target data (28R).

3. The method according to the preceding claim, in, Determining luminance value target data (28R) based on context information (K) includes: determining at least one reference image (20R) based on at least one context information (K), and determining luminance value target data (28R) from at least one reference image (20R).

4. The method according to claim 1 or 2, further comprising: Identify at least one reference image (20R) for a microscope image (20); as well as Determine the brightness value transformation (T) of the microscope image (20) (P6) to reduce the difference between the microscope image (20) and the reference image (20R).

5. A computer-based method for improving the quality of microscope images, comprising: Acquire (P1) microscope image (20); Identify at least one reference image (20R) for a microscope image (20); Determine the brightness value transformation (T) of the microscope image (20) (P6) such that the difference between the microscope image (20) and the reference image (20R) is reduced; and The brightness value transformation (T) is applied to the microscope image (20).

6. The method according to the preceding claim further comprises: Determine the luminance value data (28) of at least one reference image (20R) as the luminance value target data (28R); Determine the brightness value transformation (T) so that the brightness value data (28) of the microscope image (20) approximates the brightness value target data (28R).

7. The method according to any one of claims 4 to 6, in, Determining at least one reference image (20R) for the (P2) microscope image (20) includes: A reference image (20R) is calculated (P2b) from the microscope image (20) by an image-to-image model (M1), which is trained to calculate a quality-enhanced output image as a reference image (20R) from the input microscope image (20).

8. The method according to the preceding claim, in, The reference image (20R) calculated from the image-to-image model (M1) is registered with the microscope image (20); The value of parameter (P) of the brightness value transformation (T) is determined by minimizing the pixel difference between the microscope image (20) and the reference image (20R).

9. The method according to any one of claims 3 to 8, in, Determining at least one reference image (20R) for the (P2) microscope image (20) includes: A reference image (20R) is computed (P9) by an image generator (Gen), which receives (P8) a microscope image (20) or contextual information (K) of a microscope image (20) as input and is designed to compute an output image, wherein the image generator (Gen) is trained using a training target image with the desired brightness value attribute.

10. The method according to the preceding claim, in, A machine learning language model (LLM) generates a text description of the microscope image (20) as contextual information (K); and The text description generated by the machine learning language model (LLM) is fed into the image generator (Gen), which is designed as a text-to-image model.

11. The method according to any one of claims 3 to 10, in, Determining at least one reference image (20R) for the (P2) microscope image (20) includes: Reference images (20R) are retrieved from the database (DB) based on the microscope images (20) and / or contextual information (K) (P9').

12. The method according to any one of claims 2 to 4 or 6 to 11, in, The target data for brightness values ​​(28R) indicates the target frequency distribution of brightness values ​​(29R), and the determination of the brightness value transformation (T) makes the frequency distribution of brightness values ​​(29) of the microscope image (20) approximate the target frequency distribution (29R). or Among them, the target brightness value data (28R) indicates the target frequency distribution characteristics, especially the uniformity, entropy or diffusion of the brightness value frequency distribution, and the determination of the brightness value transformation (T) makes the frequency distribution characteristics of the microscope image (20) approximate the target frequency distribution characteristics.

13. The method according to any one of claims 2 to 4 or 6 to 12, in, Luminance value target data (28R) indicates the image statistical targets for luminance values, specifically the mean, standard deviation, contrast, and / or saturation of pixel values, as well as... The determination of the brightness value transformation (T) makes the image statistics of the microscope image (20) approximate the image statistics target.

14. The method according to any one of claims 2 to 4 or 6 to 13, in, The target data (28R) of brightness values ​​for various contextual information (K) is stored in the database (DB).

15. The method according to any one of the preceding claims, in, The context information (K) specifies one or more of the following: the microscopy method or contrast type used when taking the microscopic image (20); the sample type, sample conditions, or sample processing; The context information (K) is determined by performing image analysis on the microscope image (20) and / or querying the metadata of the microscope image (20).

16. The method according to any one of the preceding claims, in, The brightness value transformation (T) is a mapping from the brightness value (V1) to a new brightness value (V2), where the mapping is a monotonic function and is applied to all pixels of the microscope image (20).

17. The method according to any one of the preceding claims further comprises: Based on the microscope image (20) and / or contextual information, at least one range of brightness values ​​is determined, within which a brightness value transformation (T) should enhance contrast, and at least one range of brightness values ​​is determined, within which a brightness value transformation (T) should reduce contrast.

18. The method according to any one of the preceding claims, further comprising: Microscopic images were taken using a microscope (20); and The microscope image is displayed only after the brightness value transformation (T) is applied (20); in, The microscope image (20) is displayed on the screen (B) immediately after recording and brightness value transformation (T).

19. The method according to any one of the preceding claims, in, The context information (K) used to determine the brightness value transformation (T) and the microscope image with the brightness value transformation (T) applied are displayed to the user (20). Users can enter options to change the context information (K) used. If the user changes the context information (K), the brightness value transformation (T) is recalculated and applied to the microscope image (20).

20. A microscope system comprising: Microscope for image acquisition (1); and A computing unit (12) is configured to perform a computer-implemented method according to any one of the preceding claims.

21. A non-volatile, computer-readable data storage device comprising a computer program containing instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 19.

Images