Microscope system and method for validating a trained image processing model

Abstract

A microscope system comprises a microscope arranged for taking at least one microscope image (11); and a computing device (14) comprising a trained image processing model (20) arranged for computing an image processing result based on the at least one microscope image (11). The computing device (14) is arranged for verifying the trained image processing model (20) by obtaining (S1) a verification image (12) and an associated target image (16); inputting (S2) the verification image (12) into the trained image processing model (20), the image processing model computing (S3) an output image (15) from the verification image; inputting (S5) image data (28) based on at least the output image (15) and the associated target image (16) into a trained verification model (30), the verification model trained for computing an assessment (Q) for input image data (28), the assessment giving a quality dependent on the image data (28); and computing (S6) the assessment (Q) by the trained verification model (30) based on the input image data (28). Further described is a method for verifying a trained image processing model (20).

Description

Technical Field

[0001] This disclosure relates to a microscope system that uses a trained image processing model to process microscope images and a method for validating the trained image processing model. Background Technology

[0002] Automation and user comfort are playing an increasingly important role in modern microscopy systems and other measuring instruments. For example, microscopy systems should be able to image, approach, and examine samples in greater detail in a highly automated manner. To this end, panoramic or navigation maps are provided to the user, who can select a location and then automatically approach and examine that location under significant magnification via a motorized sample stage. Automatic sample identification can also be used to prevent collisions by determining the permissible range of movement of the motorized microscope components based on the identified sample or sample carrier. Certain sample structures can be automatically identified and further analyzed within the captured sample images. Biological cells can be automatically counted, or multiple partial images can be stitched together into a single image (image stitching).

[0003] For these applications, microscope images, which can be panoramic images or images of microscope samples, are analyzed using various image processing programs. Here, trained image processing models generally outperform classical algorithms and are therefore becoming increasingly prevalent.

[0004] Therefore, this type of microscope system includes a microscope configured to capture at least one microscope image, and a computing device including a trained image processing model configured to calculate an image processing result (output image) based on at least one microscope image.

[0005] Image processing models are trained using learning algorithms based on predetermined training data. These models can be, for example, deep neural networks, and specifically include CNNs (Convolutional Neural Networks) or FCNs (Fully Convolutional Networks). Using the training data, the model parameters of the image processing model, such as the terms of the convolutional matrix of a CNN, are determined during the learning process. This learning process can be monitored or not. The choice of training data, model architecture, and different characteristics of the training process all affect the quality of the resulting image processing model.

[0006] Therefore, before a trained image processing model is used in a microscope system, it should be checked against validation or test data. This involves images for which the desired image processing result (target data) is known or pre-given. The quality of the image processing model can be evaluated by comparing the target data with the image processing result calculated by the trained image processing model from the validation or test data. For a clearer understanding of the technical background and the challenges involved, refer to [reference needed]. Figure 1 .

[0007] exist Figure 1 The image 11 is schematically shown as a microscope image taken by a microscope. Microscope image 11 shows the sample carrier 7, the sample 10 to be examined, and the sample environment, such as components of a movable sample stage or components of a holder for the sample carrier 7.

[0008] Microscopic image 11 is input into a trained image processing model 20'. In this example, image processing model 20' is a segmentation model trained to compute a segmentation mask that distinguishes between sample and non-sample regions. Two possible results of image processing model 20' are shown: segmentation mask 21 and segmentation mask 22. These segmentation masks 21 and 22 are binary masks, where one of two possible pixel values ​​indicates the affiliation of the corresponding pixel with sample regions 21A, 22A, while the other pixel value indicates that the corresponding pixel is not a sample (background 21B, 22B). During the training of image processing model 20' or for quality evaluation of image processing model 20', segmentation mask 21 or 22 can be compared with a predetermined target image 16 (“ground reality”). Target image 16 is a predetermined (target) segmentation mask 24 with the correct sample region 24A and background 24B.

[0009] Comparison images 25 and 26 schematically illustrate the comparison between segmentation masks 21 and 22 and the target image 16: comparison image 25 is a superposition between segmentation mask 21 and the predetermined target image 16, while comparison image 26 is a superposition between segmentation mask 22 and the predetermined target image 16. Segmentation error 25A is shown in comparison image 25 in which sample region 22A differs from sample region 24A, which is predetermined as correct in target image 16. Similarly, segmentation error 26A is shown in comparison image 26 in the difference between sample region 22A and sample region 24A in target image 24. Figure 1As can be seen, segmentation error 25A is concentrated in a single location, resulting in a sample region 22A with a shape significantly different from the correct shape of sample region 24A based on target image 16. Conversely, segmentation error 26A forms a thin loop around sample region 22A, which substantially matches the correct shape of sample region 24A based on target image 16. In both cases, 95% of all image pixels are correctly segmented, meaning the areas of the two error regions 26A and 26B are equally large, i.e., 5% each. Commonly used evaluation metrics, such as pixel-wise accuracy of segmentation or area consistency between the segmentation mask and the target image (Jaccard coefficient, Intersection-over-Union ratio), evaluate both cases as equally good, which is schematically denoted as evaluation Q'. However, in some applications, segmentation mask 22 may be suitable while segmentation mask 21 is insufficient. Smaller edge deviations are generally more acceptable than the absence of an entire sample region. Problems often arise when the deviation is far from the correct segmentation region, which is not considered in pixel-by-pixel accuracy or cross-union ratio (CUP). For example, if multiple detailed images of some parts of a determined sample region are taken based on a sample region of a segmentation mask and then stitched into a single overall image (image stitching), this may provide correct results in the case of segmentation mask 22, while in the case of segmentation mask 21, a part of the sample may be missing during detailed inspection. Similar quality metrics, such as ORR (overall recognition rate) or ARR (average recognition rate), are not sufficiently convincing in many applications because they do not reflect the spatial and application-specific relevance of the errors that occur.

[0010] If a trained image processing model cannot deliver image processing results of sufficiently high quality, this should be determined with the least possible human cost. By targeting... Figure 1 The explanatory methods have not yet reached this point. The described measures are intended to better understand the potential problems and are based on preliminary work that is not necessarily part of the known prior art.

[0011] The applicant has described several microscope systems in DE102017111718A1, DE102017109698A1, DE102018133188A1, DE102019113540, and DE102019114012, in which image segmentation is performed using a trained model. The aforementioned problems in evaluating the trained image processing model are also more commonly seen in cases involving disparate image outputs rather than segmentation. Summary of the Invention

[0012] The purpose of this invention can be seen as providing a microscope system and a method that can perform quality checks on trained image processing models in the most accurate way possible with minimal human cost.

[0013] This objective is achieved by a microscope system having the features of claim 1 and by a method having the features of claim 2.

[0014] In the microscope system of the type described above, according to the present invention, in order to verify the trained image processing model, the computing device is configured to perform the following process:

[0015] Obtain the verification image and the associated target image;

[0016] The verification image is input into a trained image processing model, which calculates an output image from the verification image.

[0017] Image data, based at least on the output image and the associated target image, is input into a trained testing model, which is trained to compute an evaluation dependent on the input image data; and

[0018] The evaluation is calculated at least based on the input image data using a trained testing model.

[0019] Accordingly, a method for validating a trained image processing model according to the present invention includes at least the following steps: obtaining a validation image and an associated target image; inputting the validation image into a trained image processing model, the image processing model calculating an output image from the validation image; inputting image data based on the output image and the associated target image into a trained validation model, the validation model being trained to calculate an evaluation dependent on the input image data; and calculating the evaluation based on the input image data using the trained validation model.

[0020] Validating a model using machine learning allows for a significantly more robust evaluation of the quality of the computed output image, and thus the quality of the image processing model itself. Classical, simpler evaluation metrics, such as the intersection-over-union ratio explained at the beginning, only provide sufficiently accurate quality claims in certain situations. Furthermore, validating the model's training data better accounts for the vast amount of possible image content to be processed by the image processing model than classical evaluation algorithms would. For reliable evaluation through validating the model, it is crucial that the input image data consider not only the output image of the image processing model but also information about the relevant target image. Here, the target image relates to a desired outcome that the image processing model, rather than the validating model, should learn.

[0021] The validation model is particularly advantageous when image processing models should be trained repeatedly or automatically. For example, the training data for an image processing model may be frequently expanded, thus requiring a new learning process each time. Validating the model ensures that the trained image processing model delivers satisfactory quality before being used on unseen microscope images.

[0022] It is also possible that microscope users without extensive knowledge of machine learning may want to train the provided image processing model using their own training data. By using their own training data, the image processing model can be adapted to typical image types from the microscope user's experiments. By using the validation model in the manner described in the invention, the more complex manual analysis performed by the microscope user to evaluate the image processing model trained with their own training data can be eliminated.

[0023] Optional design

[0024] Advantageous variations of the microscope system and method according to the invention are the subject of the dependent claims and are explained in the following description.

[0025] Image processing model, validation image, output image, and target image

[0026] Image processing models can, in principle, be of any design, learning through machine learning and receiving at least one input image. The model architecture can, in principle, be designed in any way and may include, for example, deep neural networks, particularly one or more convolutional neural networks (CNNs).

[0027] A concrete example of an image processing model is a segmentation model, which computes an output image—a segmentation mask—from an input image. In the segmentation mask, different pixel values ​​can identify different objects or object types. Different segmentation regions are distinguished by pixel values. The segmentation mask can be a binary mask, or it can allow for more different pixel values ​​to distinguish multiple objects (types). Segmentation models can be designed for semantic segmentation, where two or more image regions are assigned a meaning (semantic). Segmentation models can also be entity segmentation models, where different objects of the same type are distinguished from each other. For example, if a sample image containing organelles is segmented and two organelles of the same type overlap or touch, entity segmentation will not output, for example, a common image region for the two organelles, but rather distinguishes between the two image regions by, for example, different pixel values. Segmentation models can also be designed to output multiple binary masks for an input image, each identifying a different object type.

[0028] More generally, image processing models can be designed to compute image-to-image mappings. Besides segmentation, this can also include denoising, where the output image has less noise than the input image. Image-to-image mappings can also be deconvolution, superresolution, or contrast-to-contrast mappings from one microscopic contrast method to another. In the latter case, graphical representations, such as phase difference or DIC (differential interferometry), are used to provide the user with familiar or easily comparable visualizations.

[0029] Image processing models can also be detection models, which output images containing detection results with their respective image coordinates. In this sense, the output images can also be formed using geometric information, such as a list of coordinates that identify the locations of the detected objects relative to the input image. The coordinates can particularly relate to bounding boxes, that is, rectangles or other shapes that enclose the detected objects within the boxes.

[0030] The values ​​of the model parameters of the image processing model are learned during training. This can be unsupervised training, in which the training data includes only the input images, or supervised training, in which an associated target image is provided as an annotation for each training / input image. The target image corresponds to the desired output of the image processing model and is also called "ground truth". The model parameter values ​​are determined iteratively, such that the computed output is as consistent as possible with the predetermined target image.

[0031] A validation image is understood to be an image that serves as input to an image processing model and for which a target image exists. For example, a validation image could be a panoramic image from a microscope. Unlike training images, validation images are typically not used in the iterative fitting of the image processing model during training, but rather to evaluate the generality or quality of the model. In traditional methods, the output image computed from the validation image is usually evaluated using the same evaluation criteria as the output image computed from the training images, such as the pixel-wise precision / intersection over union (IoU) of the segmentation results, as explained at the beginning. In the present case, the validation image is used in a different way and, in principle, can also be one of the training images used to iteratively determine the model parameter values, as will be described in more detail later.

[0032] The target image, given beforehand for the validation image, corresponds in image type to the output image of a trained image processing model. Therefore, in the segmentation model, both the output image and the target image serve as segmentation masks, in which sample regions are distinguished from non-sample regions. The target image can, in particular, be generated through manual annotation.

[0033] If an image processing model performs image-to-image mapping to increase resolution (super-resolution), it can be trained using training images, which include low-resolution images as input and higher-resolution images of the same sample as target images. The higher-resolution images can be captured using different microscopes or microscope settings, particularly different objectives, stronger illumination, or different detection characteristics.

[0034] In the case of the detection model, the image coordinates of the structure or object to be detected are given in the target image in relation to the presence of the verification image. Specifically, the verification image is used in conjunction with the target image, in which the coordinates of two or more objects to be detected, along with their corresponding object types, are pre-given. Therefore, the condition of the objects is known. This condition plays a role in the verification model, which will be described in more detail later, because certain size or positional relationships / conditions exist between some objects to be detected. For example, certain size and positional relationships typically exist between the cell nucleus, various organelles, and the cell membrane, while other relationships may predict erroneous detections.

[0035] Testing Model

[0036] The testing model is learned using a learning algorithm based on predetermined training data. Specifically, the testing model can include deep neural networks, such as one or more CNNs or FCNs.

[0037] The completed testing model should qualitatively evaluate the output of the image processing model. To this end, the image processing model first computes the output image from the validation image input to the model. A target image is pre-given for this validation image. The pairing of the target image and the output image belonging to the validation image forms image data, which the testing model uses as input data.

[0038] Image data can consist of an output image and an associated target image, meaning both images are fed together into a testing model, which then calculates a quality assessment of the output image. This quality assessment represents a measure of consistency between the output and target images.

[0039] Alternatively, a common illustration can be computed first from the output image and the associated target image. This illustration could be an image showing information from the output image and the associated target image, or the differences between them. In this case, the image data input into the trained testing model is provided through this illustration, without inputting the output image and the associated target image into the testing model. Therefore, the quality assessment of the output image can be computed based on the unique image input into the testing model.

[0040] An illustration consisting of an output image and an associated target image can be an image that shows the pixel-by-pixel difference between the output image and the associated target image. To this end, the difference between the output image and the target image can be calculated. Specifically, in the case where the segmentation mask is used as the output image and the target image, the illustration can also be calculated by adding or multiplying the output image and the associated target image pixel-by-pixel. When multiplying binary masks as the output image and the target image, the two possible pixel values ​​of the binary mask are each chosen to be non-zero, because when a zero pixel value is multiplied by pixel values ​​of other images, the result will always be zero, regardless of the pixel values ​​of the other images, thus making the image difference between the output image and the target image invisible.

[0041] In the current context, the output image of the segmentation mask is also referred to as the output segmentation mask. Similarly, the target image of the segmentation mask is referred to as the target segmentation mask. As described above, these segmentation masks are each divided into at least two distinct segmentation regions. The common representation of the output image and the associated target image can now also be a distance-transformed image. In this image, the value of a pixel gives the distance between the corresponding pixel in the output segmentation mask and the next pixel in the same segmentation region in the target segmentation mask. For example, a sample region can be segmented in both the output and target segmentation masks, where the sample regions of the two segmentation masks only partially overlap. In the distance-transformed image, the pixel value of a pixel corresponds to the distance between the corresponding pixel in the output segmentation mask (e.g., a pixel in the sample region) and the nearest pixel of the same segmentation type in the target segmentation mask (i.e., a pixel in the sample region). Therefore, the greater the image distance between corresponding structures in the output and target images, the larger the pixel value contained in the distance-transformed image.

[0042] Validate the training data of the model

[0043] The model parameter values ​​for the validation model will be determined based on the training data. This training data corresponds in type to the aforementioned image data. If the image data is formed through a common graph described, computed from the output image of the image processing model and the associated target image, then the corresponding graph is used as the training images for the validation model. The training images may each include annotations that provide relevant evaluations. The validation model thus learns to generate evaluations for the input image data.

[0044] Conversely, if the testing model is to be able to process image data that includes both the output image of the image processing model and the associated target image, then the testing model can be trained using training image data that corresponds to or is based on the output type of the image processing model. Therefore, the training image data could include the segmentation mask of the image processing model's output, the associated target segmentation mask, and corresponding evaluations of the output segmentation mask as annotations.

[0045] The annotations mentioned can be used during monitored learning. If the testing model is learning in unsupervised training, the annotations can be omitted. For example, all the training data used can correspond to positive evaluations, and the testing model can be trained using that training data as an anomaly detector.

[0046] Evaluation of the quality of image processing models

[0047] An evaluation of the output image, calculated by the image processing model against the validation images, is obtained using a trained validation model. The evaluation of the output image can, for example, indicate the number of points divided into one of several evaluation categories, or a continuous range of values. Based on the evaluation of the output image, conclusions can be inferred about the quality of the image processing model. For this purpose, multiple validation images may optionally be considered.

[0048] Therefore, multiple verification images, each with an associated target image, can be obtained. For each verification image with a corresponding associated target image, the process described for the verification image is executed, resulting in multiple evaluations output by the verification model. Based on these multiple evaluations, it is now determined whether the trained image processing model provides sufficient quality. For example, sufficient quality can be provided if a predetermined minimum number or ratio of positive evaluations for the output image is achieved.

[0049] Retraining of Image Processing Models

[0050] If the evaluation concludes that the image processing model is of insufficient quality, a new training run of the image processing model can be performed. The hyperparameters of the image processing model can be changed for the new training run. Hyperparameters can relate to, for example, the model architecture (e.g., the size or number of layers of a CNN) or the learning process (e.g., by the learning rate or the type of regularization).

[0051] In the context of this application, the verification images used may optionally be training images of the image processing model. In this case, during a new training run, the verification images for which the output images are evaluated as insufficient / defective by the model can be given greater weight. Thus, the image processing model learns in the new training run to process images more similarly to the predetermined target images in the way that the incorrectly processed verification images were.

[0052] However, alternatively or additionally, some or all of the validation images are images that were not used in the training of the image processing model to iteratively adapt the model parameter values.

[0053] General characteristics

[0054] A microscope system is understood to be a device comprising at least one computing device and at least one microscope. A microscope can, in principle, be understood as any magnifying measuring device, particularly optical microscopes, X-ray microscopes, electron microscopes, ophthalmoscopes, macroscopes, or also include magnifying image-taking devices of various designs.

[0055] The computing device can be physically designed as part of the microscope, placed independently within the microscope environment, or located at any position remote from the microscope. It can also be designed in a distributed manner and communicate with the microscope via a data connection. It can typically be formed by any combination of electronic devices and software, and particularly includes computers, servers, cloud-based computing systems, or one or more microprocessors or graphics processors. The computing device can also be configured to control the microscope camera, image acquisition, sample stage control, and / or other microscope components.

[0056] In addition to the sample camera used to capture a significantly magnified image of the sample area, a panoramic camera may also be present. Alternatively, in this case, the same camera may be involved, with different objectives or optical systems used to capture both the panoramic image and the more significantly magnified sample image. The microscope image can be directly input into the image processing model, as it was captured by the camera, or it can be processed from one or more original images before being input into the image processing model.

[0057] The computer program according to the invention includes instructions that, when executed by a computer, cause the performance of one of the described method variations. In particular, the computer program may include instructions by which the process described for a computing device can be implemented.

[0058] To clarify, let's repeat the different image names: The input image to the image processing model can be a microscope image or a verification image. The image processing model calculates an output image from it. Unless otherwise stated, in the present case, the output image refers to an image calculated by the image processing model from the verification image. The verification image can be the same as the microscope image and is characterized by having an associated target image. The target image represents the ideal output image that the image processing model should ideally calculate from the corresponding verification image. The output image of the image processing model is also called the image processing result. The output image and the associated target image (i.e., the output image is calculated from the verification image and the target image is pre-given for that verification image) together form image data, which is input into the verification model. This image data can include the output image and the target image as separate images or calculated as a unique image, which is currently referred to as a (common) illustration. The image data can also include multiple pairs consisting of the output image and the associated target image, or multiple illustrations calculated separately from the output image and the associated target image.

[0059] For ease of linguistic understanding, the singular form is generally used, which should include variations of "at least one" and "exactly one." Input of image data based on the output image and the target image (for which the model computes evaluations) can specifically mean taking at least one such input, i.e., taking two or more inputs of image data based on one output image and one target image respectively, to output an evaluation jointly for both the output image and the target image, or to output an evaluation separately for each. In the case of only one evaluation in total, this evaluation can simultaneously represent a judgment on the image processing model. The computation of the evaluation / judgment can be understood as a validation of the trained image processing model.

[0060] The method according to the invention can be used in principle for any image processing model that outputs an image or other processed result that can be graphically represented and is referred to as an output image in the present context. If the method uses stored image information, it can be performed specifically in isolation from an image-capturing device such as a microscope.

[0061] Various embodiments of the present invention exemplify the characteristics of training procedures or training data. In this regard, variations of the present invention should include, on the one hand, the implementation of the corresponding training procedures, and on the other hand, the use of the model trained in this manner (without having to perform the training procedures again), regardless of whether the described embodiments describe the characteristics of the trained model or the characteristics of the learning process.

[0062] While supervised and unsupervised learning training methods have been described for faster understanding, these variations can also be modified for partially supervised training, where only a portion of the training data is annotated. Alternatively, reinforcement learning is also possible.

[0063] When used as intended, the features of the invention described as additional device characteristics also lead to variations in the method according to the invention. Conversely, the microscope system can also be configured to perform the described method variations. In particular, a computing device can be configured to perform the described method variations and / or output control instructions for performing the described method steps. Furthermore, the computing device may include the described computer program. Attached Figure Description

[0064] Other advantages and features of the invention are described below with reference to the accompanying schematic diagrams:

[0065] Figure 1 This is a schematic diagram of image processing using a trained image processing model;

[0066] Figure 2This is a schematic diagram of an embodiment of the microscope system of the present invention;

[0067] Figure 3 The flowchart of an embodiment of the method of the present invention is illustrated schematically;

[0068] Figure 4 The training of the verification model in an embodiment of the present invention is illustrated schematically;

[0069] Figure 5 Training data for the verification model used in embodiments of the present invention are shown;

[0070] Figure 6 The flowchart of an embodiment of the method of the present invention is illustrated schematically;

[0071] Figure 7 The flowchart illustrating some steps of an embodiment of the method of the present invention is shown schematically; and

[0072] Figure 8 The flowchart of an embodiment of the method of the present invention is illustrated schematically. Detailed Implementation

[0073] Various embodiments are described below with reference to the accompanying drawings. Identical and functional components are generally identified by the same reference numerals.

[0074] Figure 2

[0075] Figure 2An embodiment of a microscope system 100 according to the present invention is shown. The microscope system includes a computing device 14 and a microscope 1, which in the illustrated example is an optical microscope, but in principle could be a different type of microscope. The microscope 1 includes a support 2 by which other microscope components are held. Specifically, it may include: an objective turret or objective mount 3, on which, in the illustrated example, an objective 4 is mounted; a sample stage 5 having a holding frame 6 for holding a sample carrier 7; and a microscope camera 8. If the objective 4 is rotated into the microscope beam path, the microscope camera 8 receives detection light 7 from one or more samples held by the sample carrier to capture an image of the sample. The sample can be any object, fluid, or structure. The microscope 1 also includes a panoramic camera 9 for capturing a panoramic image of the sample environment. The panoramic image can thereby specifically show the sample carrier 7 or a portion thereof. The field of view 9A of the panoramic camera 9 is larger than the field of view when capturing the sample image. In the illustrated example, the panoramic camera 9 observes the sample carrier 7 through a mirror 9B. The mirror 9B is arranged on the objective mount 3 and can be selected to replace the objective 4. In a variation of this embodiment, the reflector or another deflecting element may be arranged in other positions. Alternatively, the panoramic camera 9 may be arranged so that it directly observes the sample carrier 7 without the reflector 9B. Although in the example shown, the panoramic camera 9 observes the sample carrier 7 from above, alternatively, the panoramic camera 9 may be arranged so that it sees the bottom surface of the sample carrier 7. In principle, this is done by selecting the objective mount 3 to capture a panoramic image.

[0076] In the present context, a microscope image can be understood as referring to a panoramic image or a sample image, as described above. A microscope image can correspond to the raw data captured or be formed simply through further processing of the raw data. The computing device 14 includes a computer program 80 having a trained image processing model for processing at least one microscope image.

[0077] Image processing models should process input images into usable outputs as reliably as possible. Incorrect results can render experiments unusable, produce image artifacts that frustrate or confuse users, or lead to incorrect microscope controls or movements that could result in collisions and component damage.

[0078] Therefore, an image processing model should be validated before it is used in normal operation. If new training of the image processing model is performed more frequently, for example with supplemental training data, the manual workload of validation should be kept to a minimum. For example, it may be optionally stipulated that microscope users want to train the image processing model using their own microscope images as training data suitable for their own experiments. Before the image processing model is actually put into use, validation should ensure that the image processing model functions correctly. This is done by validating the learned model, which will be described in more detail with reference to the following figures.

[0079] Figure 3

[0080] Figure 3 The flowchart of one embodiment of the method of the present invention is illustrated schematically for evaluating a trained image processing model. These flowcharts can be implemented using... Figure 1 It is implemented using computing devices or computer programs.

[0081] First, in step S1, a verification image 12 is obtained. This verification image can be a microscopic image or can be calculated from one or more microscopic images. The verification image 12 can be loaded from memory or directly from... Figure 1 The image was taken using a microscope. In the example shown, verification image 12 is a microscope image showing sample carrier 7 with sample 10.

[0082] In step S2, the verification image 12 is input into the trained image processing model 20, which calculates the output image 15, step S3. For example, the image processing model 20 is a segmentation model that calculates a segmentation mask 21 as the image processing result or output image 15. In the segmentation mask 21, one pixel value identifies the sample region 21A, while another pixel value represents the background 21B.

[0083] A target image 16 is pre-defined for the verification image 12. This target image represents the ideal result that the output image 15 of the image processing model 20 should approach as closely as possible. In the example shown, the target image 16 is a target segmentation mask 24 with sample region 24A and background 24B.

[0084] Under the condition of high quality of image processing model 20, the sample region 21A of segmentation mask 21 should correspond as precisely as possible to the sample region 24A of target image 16 or as precisely as possible to the image region of sample 10 in verification image 12. This cannot be satisfactorily determined by simple evaluation measures. As similar to the opening section Figure 1As explained, for example in the case of a flawed segmentation mask 21, the area consistency between the sample region 21A of the segmentation mask 21 and the sample region 24A of the target image 16 can be very high. For example, even in the case of insufficient segmentation results, more than 95% of all pixels can be correctly classified.

[0085] To examine segmentation mask 21, or more generally, output image 15, a common illustration 27 is first calculated from output image 15 and target image 16 in step S4. Illustration 27 is an example of general image data 28 based on output image 15 and associated target image 16. To form illustration 27, output image 15 and target image 16 can be superimposed, for example, by adding or subtracting them pixel by pixel. Common illustration 27 shows segmentation errors or general image differences 27A that make the differences between output image 15 and target image 16 identifiable. Whether image difference 27A is significant or negligible can depend on many factors, such as its shape, its size, its size relative to the segmented sample region 21A, its shape and / or size relative to the segmented sample region 21A, its absolute position in the image, and / or its relative position in the image relative to the segmented sample region 21A. In the case of objects different from the sample region, other features may be particularly important.

[0086] Although it is nearly impossible to handle these features, and especially the diversity of possible features, through feasible evaluation criteria, a reliable evaluation can be performed using a trained test model 30.

[0087] In step S5, the testing model 30 obtains illustration 27 as the input image. The testing model 30 is composed of a deep neural network, which is trained based on training data T to calculate a score Q for the input image. In step S6, it outputs an evaluation Q for the output image 15. The training data T includes multiple images or illustrations, each pre-given with an annotation A that provides a good or bad evaluation. Thus, the testing model 30 is trained to generate an evaluation Q corresponding to annotation A for unknown input images based on the image content of the input image.

[0088] The training process of the test model 30 will be described below with reference to the attached figures.

[0089] Figure 4

[0090] Figure 4The training process of the testing model 30 is illustrated schematically. This training process is part of some method variations of the present invention. Other variations of the present invention use the testing model 30 trained in this manner, and the training step itself is not part of the claimed method steps.

[0091] Training data T is provided, which consists of multiple training images T1-T8, each with its own annotation A. Annotation A provides an associated evaluation for each training image T1-T8. In the example shown, this can be a good or bad evaluation, and it is also possible to have, in principle, a higher number of evaluation categories or value data that is, in principle, arbitrarily refined within a range of values.

[0092] In the example shown, the training images T1-T8 are formed as illustrated in Figure 27, that is, by superimposing or computing the output image of the image processing model to be evaluated with a predetermined target image of the image processing model. However, Figure 27 can also be generated more generally from the output images of other image processing models. In the example shown, they can be formed by superimposing any segmentation mask and an associated target / segmentation mask assumed to be true.

[0093] Training images T1-T4 display some illustrations, for which annotation A pre-defines a positive evaluation. Conversely, for training images T5-T8, negative evaluations are pre-defined via annotation A.

[0094] The advantages of traditional quality standards for evaluating segmentation results are discussed below, using the reference training image T5. The target image belonging to T5 shows an elongated structure, in which only the right end of the structure is correctly identified in the associated output image, while the rest of the elongated region (the area filled in black in T5) is completely missing. In this case, commonly used quality standards such as ORR or ARR provide high quality because the missing elongated structure comprises only a few pixels and therefore only a few pixels are incorrectly segmented / classified. Conversely, such cases can be identified and evaluated with high reliability after training with training images T1-T8.

[0095] Training images T1-T8 are input into a testing model 30 to be trained, which computes an output Q based on initial model parameter values. In this example, a positive or negative evaluation is computed as the output Q for each training image. A predetermined loss function L describes the difference between the output Q computed with the current model parameter values ​​and the annotation A. An optimization function O iteratively minimizes the loss function L, to which the optimization function is iteratively adapted to the values ​​of the model parameter values, for example, through gradient descent and backpropagation.

[0096] Alternatively, during training, contextual information i can be input along with training images T1-T8. This allows a model to be learned that incorporates the input contextual information into the calculation of the evaluation Q. Contextual information i can, for example, relate to the sample type, providing information about the possible or impossible shapes, sizes, and quantities of sample regions in the images. Furthermore, contextual information can relate to the type of experiment performed, the equipment setup, or the type of sample carrier.

[0097] Annotation A can also be evaluated in a more sophisticated way. For example, multiple evaluations can be pre-defined as annotations for some or all of the training images. For instance, multiple evaluations for the same training image can be manually pre-defined by different users. Such broader estimation may increase robustness.

[0098] In a similar manner, the target image can also be based on estimates from multiple users. For example, multiple users can pre-define target images for the same training / validation image and use the average of these multiple target images in the training described in the present context.

[0099] Figure 4 The monitored learning process is illustrated. In other modifications, an unmonitored learning process without predetermined annotations A can be used. For example, a testing model 30 can be trained for anomaly detection using unmonitored learning. Therefore, all training images used can correspond to a positive evaluation. If the trained testing model receives an input image that deviates from the distribution of the training images used, an anomaly is identified and interpreted as a negative evaluation.

[0100] Figure 5 and Figure 6

[0101] Figure 5 The diagram illustrates training data T, which can be used to replace... Figure 4 The training data shown is in [the image]. Figure 5 In this case, instead of calculating a common graph consisting of the output image of the image processing model and the associated target image, there are corresponding images as pairs. The training image T1A corresponds to the validation image or input image used for the image processing model, while the training image T1B is the associated target image.

[0102] The training data also includes other paired training images T2A, T2B and T3A, T3B. An annotation A is provided beforehand for each paired training image, which provides an evaluation, such as an evaluation of the similarity between training image T1A and its associated training image T1B.

[0103] The input to the testing model is not formed from a single training image, but rather from paired training images T1A, T1B, or T2A, T2B, or T3A, T3B. In this way, as in the previous embodiments, the testing model can learn to evaluate unseen images.

[0104] The use of the testing model 30 trained in this way in Figure 6 The diagram is shown schematically. (And) Figure 3 The difference in the implementation scheme is that a common illustration consisting of the output image 15 and the associated target image 16 is not calculated. Instead, the output image 15 and the associated target image 16 are input together as image data 28 into the testing model 30, and the testing model 30 thereby calculates the evaluation Q.

[0105] according to Figure 3 and Figure 4 The use of common diagrams reduces the complexity of the problem. In some cases, less training data may be needed and better generalization can be achieved. Conversely, according to Figure 5 The use of two separate images can be employed in certain situations through different image processing models, without any additional adaptation other than selecting the training data accordingly.

[0106] Figure 7

[0107] Figure 7 Partial steps of a method variation according to the invention are shown to illustrate that the image processing model 20 need not be a segmentation model and that the microscope image or verification image used as input to the image processing model 20 can also be a pre-processed image. Figure 7 In this example, a raw image 18 is first captured or processed, showing, in this example, a sample carrier 7 held between two movable retaining frame clips 6A and 6B of a retaining frame. From the raw image 18, a segmentation mask 19 is first calculated, in which the two retaining frame clips 6A and 6B are distinguished from the background. The segmentation mask 19 is formed as input to the image processing model 20, whereby the segmentation mask represents the microscope image or verification image 12 in the currently used terminology.

[0108] Image processing model 20 is a detection model in this example. The detection model outputs the image coordinates of the detected object 6C, and in this example, it outputs the image coordinates of the inner corners of the frame clamps 6A and 6B.

[0109] The image coordinates of the object 6C being tested represent the output image 15, which should be evaluated using the testing model in the manner already described. For easier understanding, Figure 7The superposition of the detected object 6C with the verification image 12 is shown, but superposition is not mandatory. The number of detected objects 6C can vary depending on the positioning of the frame, where only certain conditions of the detected objects 6C may actually occur. By means of a target image pre-given for the output image 15 or the verification image 12 as the basis, the verification model can thus learn to evaluate the conditions among the detected objects 6C and thus distinguish the correct image processing results from the incorrect results of the image processing model 20.

[0110] Figure 8

[0111] Figure 8 A flowchart of a method variation according to the present invention is shown, illustrating the purpose of using the test model 30.

[0112] Figure 8 The steps in this process are suitable for new training of image processing model 20 and can be performed at the microscope manufacturer or by the microscope user. For example, new training of the existing image processing model 20 may be necessary if new training images are added, such as training images for different sample types, new sample carrier types, or contrast methods for another imaging method. Before this, human experts evaluate the quality of the learned model, a task currently undertaken by testing model 30.

[0113] First, in step S10, new training images are obtained for the image processing model 20. The training images may supplement or replace existing training images used for the image processing model 20, or they may be the initial training images used for the image processing model 20.

[0114] In step S11, the image processing model 20 is trained using the training images, that is, the model parameter values ​​of the image processing model 20 are determined based on the training images.

[0115] Next, in step S12, the output image calculated by the trained image processing model 20 is evaluated, for example, as described with reference to the previous figures. Specifically, multiple new training images or previously existing training images can be input into the trained image processing model 20. The input images can be the same as the images used in the iterative fitting of the model parameter values ​​for the image processing model 20, or they can be a separate set of images not used in the iterative fitting. The output images calculated by the image processing model 20 from these images (validation images) are fed together with predetermined target images to a testing model 30, which calculates an evaluation for each of these output images.

[0116] In step S13, the judgment of the trained image processing model 20 is determined by these evaluations of the output images. For example, at least a predetermined proportion of all evaluations of the output images must be positive / good, such that the trained image processing model 20 is classified as qualitatively good. Alternatively, steps S12 and S13 can also be implemented together by the testing model 30. It is not absolutely necessary for the testing model 30 to output evaluations separately for each of the various output images considered; rather, it is sufficient if the testing model 30 considers these output images and the associated target image and outputs only one evaluation of the testing model 20 in general.

[0117] In step S14, it is checked whether the model 20 is evaluated as positive or negative. If the evaluation is negative, proceed to step S15. In this step, the hyperparameters of the training can be changed, and then new training is performed in step S11. Alternatively, a warning instruction or a command to change or supplement the training images can be issued to the user, and then proceed to step S10.

[0118] Conversely, if the judgment in step S14 is positive, the image processing model 20 is released for further use. In step S16, the image processing model 20 can be used to process images that were not seen during the training of the image processing model 20. In this intended use of the image processing model 20, the testing model 30 is no longer used.

[0119] The described embodiments are purely illustrative and modifications to these embodiments are possible within the scope of the appended claims.

[0120] List of reference numerals

[0121] 1. Microscope

[0122] 2 brackets

[0123] 3 Objective mount

[0124] 4. Microscope Objectives

[0125] 5 sample stages

[0126] 6. Maintain the frame

[0127] 6A, 6B retain the frame clip

[0128] 6C test results / Keep the inner corner of the frame clip

[0129] 7 Sample carriers

[0130] 8 microscope cameras

[0131] 9 panoramic cameras

[0132] 9A panoramic camera's field of view

[0133] 9B reflector

[0134] 10 samples

[0135] 11 Microscopic images

[0136] 12 Verification Images

[0137] 14 Computing devices

[0138] Output image of 15 image processing models

[0139] 16 target images

[0140] 20 image processing models, especially segmentation models

[0141] 20' Image processing models, especially segmentation models

[0142] 21, 22 segmentation mask

[0143] 21A and 22A are sample regions in segmentation mask 21 or 22.

[0144] 21B and 22B in the background of segmentation mask 21 or 22

[0145] 24-target segmentation mask

[0146] 24A in the sample region of the target image

[0147] 24B in the background of the target image

[0148] Comparison of images 25 and 26

[0149] 25A and 26A are incorrectly segmented.

[0150] 27 (Common) Diagram

[0151] 27A Image Difference

[0152] 28 Image data based on output image 15 and associated target image 16

[0153] 30 test model

[0154] 80 computer programs

[0155] 100 Microscope System

[0156] A note

[0157] i Context Information

[0158] L loss function

[0159] O optimization function

[0160] Q outputs the image for evaluation 15 / test the model's output;

[0161] Q' is not an assessment based on the present invention.

[0162] S1-S6 Steps / processes of the method variation of the present invention

[0163] S10-S16 Steps / processes of the method variation of the present invention

[0164] T training data / training image data

[0165] Training images T1-T8

Claims

1. A microscope system, comprising: A microscope, the microscope being configured to take at least one microscope image (11). and Computing device (14), the computing device including a trained image processing model (20) configured to calculate image processing results based on the at least one microscope image (11); The computing device (14) is configured to verify the trained image processing model (20) in the following manner: Obtain (S1) the verification image (12) and the associated target image (16); The verification image (12) is input (S2) into the trained image processing model (20), and the image processing model calculates (S3) the output image (15) from the verification image. Its features are, Based on the input, image data (28) generated at least from the output image (15) and the associated target image (16), an evaluation (Q) of the image processing model (20) is calculated (S6) by a trained testing model (30), wherein the testing model is trained to calculate the evaluation (Q) for the input image data (28), the evaluation (Q) indicating a quality of the image processing model (20) that depends on the image data (28).

2. A method for validating a trained image processing model (20), characterized in that, The method includes at least the following processes: Obtain (S1) the verification image (12) and the associated target image (16); The verification image (12) is input (S2) into the trained image processing model (20), and the image processing model calculates (S3) the output image (15) from the verification image. Based on the input, image data (28) generated at least from the output image (15) and the associated target image (16), an evaluation (Q) of the image processing model (20) is calculated (S6) by a trained testing model (30), wherein the testing model is trained to calculate the evaluation (Q) for the input image data (28), the evaluation (Q) indicating a quality of the image processing model (20) that depends on the image data (28).

3. The method according to claim 2, Its features are, The image data (28) input into the trained testing model (30) includes the output image (15) as a separate image and the associated target image (16).

4. The method according to claim 2, further comprising: The illustration (27) is calculated from the output image (15) and the associated target image (16); The image data (28) input into the test model (30) is a diagram (27) of the calculation. The illustration (27) consisting of the output image (15) and the associated target image (16) is an image showing the pixel-by-pixel differences between the output image (15) and the associated target image (16).

5. The method according to claim 2, Its features are, The output image (15) is an output segmentation mask (21), and the target image (16) is a target segmentation mask (24). The image data (28) shows the segmentation error or image difference (27A) between the output segmentation mask (21) and the target segmentation mask (24); the trained testing model (30) evaluates whether the image difference (27A) is relevant based on at least one of the following: the shape of the image difference (27A), the shape and / or size of the image difference (27A) relative to the segmentation region (21A), or the relative position of the image difference (27A) relative to the segmentation region (21A).

6. The method according to claim 2, Its features are, The output image (15) is an output segmentation mask (21), and the target image (16) is a target segmentation mask (24). They are each divided into at least two distinct partitioned regions (21A, 21B; 24A, 24B). The figure (27) is calculated from the output image (15) and the associated target image (16); The illustration (27) consisting of the output image (15) and the associated target image (16) is a distance-transformed image in which the pixel value gives the distance between the corresponding pixel of the output segmentation mask (21) and the next pixel of the same segmentation region in the target segmentation mask (24). The image data (28) input into the test model (30) is the calculated illustration (27).

7. The method according to any one of claims 2 to 5, wherein the method further comprises: Obtain multiple verification images (12) that are associated with the target image (16); The process mentioned in claim 2 is performed on each of the verification images (12) having a separately associated target image (16) so that the verification model (30) outputs a plurality of evaluations (Q) (S12), and based on the evaluations (Q), it is determined (S14) whether the trained image processing model (20) provides sufficient quality.

8. The method according to claim 7, Its features are, Based on the judgment, a new training run of the image processing model (20) is performed by means of the changed hyperparameters of the image processing model (20).

9. The method according to claim 2 or 3, Its features are, The trained testing model (30) has been trained using training image data, which includes images as input images that correspond to the type of the output image (15) of the trained image processing model (20). The trained testing model (30) is trained via unsupervised training; in this unsupervised training, all images corresponding to the type of the output image (15) of the trained image processing model (20) correspond to positive evaluation results.

10. The method according to any one of claims 2 to 5, Its features are, The output image (15) is an output segmentation mask (21), and the target image (16) is a target segmentation mask (24). The trained testing model (30) has been trained using training image data, which includes the segmentation mask (T1A-T3A) and the corresponding target segmentation mask (T1B-T3B) as input data, and the evaluation results of each segmentation mask (T1A-T3A) as annotation (A).

11. The method according to any one of claims 2 to 4, Its features are, The image processing model (20) calculates image-to-image mappings, in which denoising, deconvolution, resolution enhancement, or mappings from one microscopic contrast method to another are computed. The testing model (30) is trained using training image data (T) that corresponds to the output type of the image processing model (20).

12. The method according to any one of claims 2 to 4, Its features are, The image processing model (20) is or includes a detection model that provides a detection result (6C) with corresponding image coordinates as an output image (15), wherein the associated target image (16) provides a predetermined detection result with corresponding image coordinates.

13. The method according to any one of claims 2 to 5, Its features are, The trained testing model (30) is trained using training image data (T) corresponding to the image data (28) and including multiple training image pairs (T1A, T1B; T2A, T2B; T3A, T3B) and annotations (A) indicating the corresponding evaluation value (Q) of the similarity between each training image (T1A, T1B; T2A, T2B; T3A, T3B) within the corresponding training image group.

14. A non-transitory computer-readable medium comprising a computer program having instructions which, when executed by a computer, cause to perform the method according to any one of claims 2 to 5.

Images