Microscope systems and methods for classifying interchangeable components of microscopes

Abstract

This invention relates to a method for classifying interchangeable parts (9) of a microscope, the method comprising: obtaining a microscope image (20) in which at least a partial image of the interchangeable part (9) is formed; calculating a processed image (30) from the microscope image (20); and performing an image evaluation on the processed image (30) to classify the interchangeable part (9). The calculation of the processed image (30) includes removing image content or surface features of objects (11, 12, 13) from at least a region of the microscope image (20) converted into the processed image (30). A corresponding microscope system is also described.

Description

Technical Field

[0001] This invention relates to microscope systems and a method for classifying interchangeable components of microscopes by means of image processing. Background Technology

[0002] For intuitive and at least partially automated operation of microscopes, the automated detection of interchangeable parts attached to or placed on the microscope is an important aspect. Interchangeable parts can be sample carriers, holders for sample carriers, or DIC slides (slides used for differential interference contrast). The evaluation of overview images is increasingly automated to identify existing interchangeable parts.

[0003] A general-purpose microscope system includes a microscope comprising at least one interchangeable component and configured to acquire a microscope image, wherein the interchangeable component is at least partially imaged. A computing device is configured to calculate a processed image from the microscope image and evaluate the processed image to classify the interchangeable component.

[0004] A general method for classifying interchangeable parts of a microscope includes, in a corresponding manner, at least the following steps: obtaining a microscope image, wherein the interchangeable parts are at least partially imaged; calculating a processed image from the microscope image; and evaluating the processed image to classify the interchangeable parts.

[0005] Microscopic images can be captured, in particular, by a panoramic camera, so that in addition to the area where the sample to be examined is placed, the surrounding microscope components are also visible, including interchangeable components to be classified. To process the microscope images, adjustments such as contrast or brightness are typically performed. The processed images obtained from them are now evaluated in detail to classify the interchangeable components, i.e., to identify the type or model of the interchangeable components. This is exemplarily referenced... Figure 1 Please provide an explanation. The applicant described a similar approach in DE 102017109698 A1.

[0006] Figure 1 The schematic diagram illustrates the flow of a conventional method for classifying interchangeable parts 9, and in this example, the retainer 10. Microscopic image 20 shows the retainer 10 with a sample recess 14 in which the sample or sample carrier will be positioned. Additionally, portions of the shape of the retainer 10 and surface features of the retainer 10 can be identified.

[0007] The classification of cage 10 is achieved using a classification CNN 40' (Convolutional Neural Network). The classification CNN 40' has been trained with the aid of training data displaying various known types of cages. Therefore, it is able to classify the input microscope image 20 as follows: what kind of cage 10 is imaged in the microscope image 20. The classification CNN 40' outputs output 45', which ideally indicates the correct cage 10 from several possible cage types.

[0008] During training, the classification CNN 40' learns largely independently which image contents and how these contents indicate the specific type of cage 10. Particularly relevant here are the details of the cage 10, magnified in images 21, 22, and 23. These include labels 11, such as manufacturer name, model number, or serial number; screws 12; clips and other objects on the cage; and scratches 13 on the cage 10. Labels 11 and screws 12 may differ between different types of cages, both in their arrangement and their appearance, thus they are suitable distinguishing features in themselves. However, labels 11 and screws 12 can be changed by the cage manufacturer at any time without altering the cage's function or model name. In these cases, the classification CNN 40' must be retrained with the aid of new training data showing updated cages from the manufacturer. Another problem may arise from scratches 13 on the cage, especially if the training data includes multiple images of the same cage. In this case, the classification CNN 40' learns to identify the individual attribute of a particular cage, namely its scratches 13 or stains. This affects the identification of cages of the same type, whether they have scratches or not. To reduce problems caused by scratches and other individual surface features of the cages, Figure 1 The classification CNN 40' in this example requires a relatively large amount of training data. However, even with a large amount of training data, the problem of relatively poor generalization still exists. If the microscope image 20 differs significantly from the training data (e.g., because the manufacturer changed the arrangement of screw 12), the classification reliability of output 45' indicating the correct type of cage decreases.

[0009] replace Figure 1 The classification CNN 40' shown as an example also uses other machine learning models or classical algorithms for classification, which also have problems with generalization or classification reliability. Summary of the Invention

[0010] The present invention may be considered to include a microscope system, a method, and a computer program that allow for the classification of interchangeable parts of a microscope in a reliable and simple manner.

[0011] This objective is achieved by the method according to the invention and the microscope system according to the invention.

[0012] In the above-described method, according to the present invention, the calculation of image processing includes removing image content of object or surface features, at least in the region of the microscope image that is converted into the processed image.

[0013] In a corresponding manner, in a microscope system, a computing device for calculating and processing images is configured to remove image content of object or surface features, at least in the area of ​​the microscope image that is converted into a processed image.

[0014] The computer program according to the invention includes instructions that, when executed by a computer, cause the execution of the method according to the invention.

[0015] Therefore, the image assessment used to classify interchangeable parts is not based on microscopic images but on a less complex display calculated from them. In the example described at the beginning, the objects from which the image content is removed can be, in particular, screws and labels on interchangeable parts, or more generally, any element on interchangeable parts, such as one or more of the following: labels, markings, stickers, scratches, clips, screws, and stains. Surface features or textures can particularly involve color, material selection, matte or glossy surfaces, reflections, and scratches.

[0016] According to the present invention, by using processed images with reduced complexity instead of microscope images or images with the same content for classification, the risk of misclassification caused by irrelevant appearance features of interchangeable parts in microscope images can be reduced or eliminated.

[0017] Optional design

[0018] Advantageous variations of the microscope system, computer program, and method according to the invention are the subject of the dependent claims and are explained in the following description.

[0019] Image processing computation

[0020] Calculating the processed image from a microscope image can be achieved using a trained machine learning model, particularly a CNN (Convolutional Neural Network) or other deep learning neural networks. A CNN can be a segmentation CNN that performs microscope image segmentation.

[0021] When segmenting a microscope image, the segmentation mask records which image regions represent interchangeable parts and which do not. This resulting segmentation mask can represent the processed image or can be further computed to produce a processed image. The segmentation mask can be a binary mask, where one pixel value identifies the image point belonging to the interchangeable part, and another pixel value indicates that the image element with that pixel value does not belong to the interchangeable part. Therefore, the segmentation mask shows the shape of the interchangeable part, rather than representing surface features or objects / elements on the interchangeable part. Instead of a binary mask, the segmentation mask can also specify more than two values, for example, to distinguish several interchangeable parts in the same image or to distinguish different components of the same interchangeable part. The segmentation mask can be a two-dimensional image with the same resolution as the microscope image. Alternatively, the segmentation mask can also correspond to only a portion of the microscope image and therefore have a lower resolution. Furthermore, the segmentation mask can be represented using vector graphics or tabular data.

[0022] Alternatively, image processing can be calculated by cropping or trimming image regions of objects, such as the image region of a screw. Image pixels within the cropped area have no value, have the same value, or are filled with a predetermined pattern, while surrounding image pixels outside the cropped area continue to carry image information and are converted into a processed image. This does not correspond to simple cropping of microscope images, as is the case when using ROIs (Regions of Interest) in the prior art. This is because, in the current case, the cropped image region (which, for example, shows a screw or other interfering object) is located within the image portion being converted into a processed image. When using an ROI, the cropped image region is located outside the image portion that continues to be used. Therefore, it is not possible to achieve the effect of removing surface features and interfering objects such as screws from the illustration of interchangeable parts using ROIs. However, in variations of the invention, in addition to the described steps, cropping of ROIs known in the prior art can be performed. In the current case, the external shape of the interchangeable parts remains untouched and thus still identifiable. An input tool can be optionally provided, which the user can use to select or mark the image region to be cropped.

[0023] To compute the image, alternatively, image regions of objects in a microscope image, such as those of a screw, can be inpainted based on adjacent image content. This creates a natural image impression with a smooth transition to adjacent image regions. Compared to the cropping described above, inpainting advantageously avoids creating any hard edges that could impair the subsequent classification process. Inpainting can be performed using a machine learning model trained specifically for this purpose.

[0024] The aforementioned implementations and other variations can be achieved by performing an image-to-image imaging machine learning model. The machine learning model can be trained on training images with associated target images, where objects or surface features in the target images are removed from the training images. In this way, the machine model learns which objects and how they can be removed. In another variation, a machine learning model including an autoencoder is used to compute the processed image based on the microscope image. The autoencoder is trained to generate an output image that is as similar as possible to the input image. To do this, for example, a loss function that detects the differences between the input and output images can be minimized. Here, the autoencoder is formed by a neural network where the layers have a minimum number of neurons (bottleneck). This bottleneck layer determines the level of detail that the autoencoder can image. By reducing the number of neurons in the bottleneck layer, the level of detail in the processed image can be reduced. This allows for the effective removal of small objects or interfering surface features such as scratches and screws. The size of the bottleneck layer is chosen such that surface features are removed from the input image (microscope image) in the existing training or testing data, while the shape or size of interchangeable parts remains unchanged in the processed image. In addition to or alternative to setting the bottleneck, the level of detail that the autoencoder can image can also be set via an associated loss or gain function. For example, higher spatial / image frequencies can be penalized in the loss function. Instead of an autoencoder, a similar effect can be achieved if the neural network generates image-to-image imaging while simultaneously converting the input image into any intermediate representation that suppresses unwanted features. This can be particularly determined by the size of the bottleneck layer, as described for autoencoders.

[0025] In variations of the invention, processed images showing only the shapes of interchangeable parts are generated by removing surface features and the aforementioned image content of one or more objects. This is, for example, the case of generating binary masks using a segmentation machine learning model. The advantage here is that the segmentation machine learning model can segment across various categories of interchangeable parts (e.g., different models of fixtures) without knowing the current category. The segmentation machine learning model, or any other machine learning model used, is preferably trained with training images covering several or all categories, which should then be distinguished by a classification model. By being able to use segmentation information across categories, the amount of training images required is reduced, which is necessary for training robust segmentation CNNs across categories.

[0026] Classification

[0027] It can be configured that, in the image evaluation of the processed image used to classify interchangeable parts, (only) the shape of the interchangeable parts is considered, while the texture of the interchangeable parts is disregarded. To this end, as described above, the texture can be removed from the image information.

[0028] A trained classification machine learning model can perform image evaluation on the processed image to classify interchangeable parts. The machine learning model can, in particular, be a CNN designed to classify the input image and referred to herein as a classification CNN. Compared to conventionally used classification machine learning models, this model can be trained with training data computed from the corresponding microscope images in the same manner as the processed image described. Specifically, the training images can therefore be segmentation masks or binary masks. The advantage of this reduced complexity in training images lies in better generalization: if interchangeable parts belonging to a known category but appearing different in the microscope image are used, for example because the manufacturer of the interchangeable parts has changed details such as screws, labels, or stickers, this does not change the segmentation mask / binary mask computed from the microscope image, allowing the classification CNN to correctly identify the interchangeable parts. Furthermore, different lighting environments only affect the microscope image but generally do not affect the segmentation mask computed thereby. This is beneficial for training classification machine models because, unlike paragraph 0079 from DE 102017109698A1, among other things, the lighting environment and other surrounding conditions or stains do not need to be reproduced from the training images.

[0029] To increase the robustness of the classification model, simulation programs can be used to generate processed images (specifically, segmentation masks). This is particularly meaningful if there are movable or variable parts on the interchangeable parts to be classified, whose shapes can be altered by the segmentation mask. For example, different configurations of interchangeable parts can be simulated using CAD models in the simulation program. Since only the resulting segmentation mask is needed, complex simulations (rendering) of the appearance with natural lighting are unnecessary. This process is especially advantageous if the available training dataset contains many classes, but only a relatively small number of images per class, and these images do not cover all possible variations of the segmentation mask. The simulated segmentation mask can be used as training data for a classification machine learning model, which can then classify interchangeable parts in the processed images.

[0030] Additionally, correlation information can be used to classify interchangeable parts. Correlation information can relate to microscope settings, such as the filters, light source, or detection properties used. This allows for the inference of potentially usable interchangeable parts or the exclusion of certain types of interchangeable parts. Further correlation information can be sample attributes, regarding the type of sample or sample carrier identified in the microscope image or otherwise. Knowing the type of sample carrier is particularly helpful if the interchangeable part is a cage. Optionally, the aforementioned information can be obtained by evaluating any labels that may be present on the sample carrier from the microscope image. Correlation information, along with the processed image, can form the input to a classification CNN.

[0031] Machine learning models can both compute processed images from microscope images and perform image evaluations on processed images to classify interchangeable parts. Alternatively, a single machine learning model can be used for these steps. The described machine learning models, particularly segmentation CNNs and subsequent classification CNNs, can be combined / connected and trained together for this purpose. If a single machine learning model or CNN is used, the information in the processed image forms intermediate layers in the neural network. Here, the intermediate layers may optionally contain information in the processed image in an encoded form, that is, not in particular as a 2D matrix of image points, but as a tensor, which is first converted into a 2D image by the decoder.

[0032] In principle, classification can also be performed using classic classification algorithms without the need for machine learning models. The classification algorithm extracts geometric properties from the processed image and compares these properties with reference values ​​to determine the category of interchangeable parts. Geometric properties can be, for example, the shape and size of the perimeter, peripheral segments, or openings of the interchangeable part as depicted in the processed image.

[0033] General characteristics

[0034] Interchangeable parts can, in principle, be any microscope component that is not permanently fixed in position to the microscope, but rather that is interchangeable with other microscope components or whose position or configuration is variable. Interchangeable parts can be, for example, sample carriers, sample carrier holders, samples, sample stages, microscope objectives, illumination modules, filters or prisms, condensers, filter inserts, particularly DIC slides, components for polarization contrast recording, gratings, or other light modulators. Objectives held on the objective changer and not currently in the microscope beam path can also be understood as interchangeable parts. For better understanding, the classification of individual interchangeable parts in the processed image is described in various implementation variations. However, in principle, multiple interchangeable parts can also be shown and classified in the same processed image.

[0035] Classification is understood as representing the categories / types of interchangeable components. Different categories of sample carriers can be, for example, microtiter plates, petri dishes, and chamber slides, and / or different models of microtiter plates, petri dishes, and chamber slides. If the interchangeable component is a sample, different sample types can be identified, particularly material samples, embedded sections, or printed circuit boards / PCBs. If the interchangeable component can be used in different arrangements or configurations, these arrangements can be specified as being identified as different categories in the classification.

[0036] A microscope system is understood to be a device comprising at least one microscope and a computing device. The computing device may be physically designed as part of the microscope, or it may be arranged separately around the microscope. Alternatively, the computing device may be designed in a distributed manner and communicate with the microscope via a data connection. A microscope can be understood in particular as an optical microscope, X-ray microscope, electron microscope, microscope, or other magnifying image recording instrument configured for recording images (microscope images). According to embodiments of the invention, the image recording process may be part of the method, or the method may begin by loading an existing microscope image.

[0037] Microscope images can be overview images captured by the microscope's overview camera. An overview camera can exist in addition to the camera that observes the sample through the microscope objective (sample camera). However, in principle, microscope images can also be images captured by the sample camera, especially if a microscope objective with a low imaging ratio is used. Typically, microscope images can also be captured by a camera not attached to the microscope holder, such as using a handheld device like a smartphone.

[0038] The computing device can be specifically configured to perform variations of the described methods. The computing device can be connected to or assigned to a particular microscope, or used to control the microscope. The connection can be wired, via the Internet, or other networks. In particular, the computing device may include a personal computer or one or more (graphics) processors that are also part of the microscope. Alternatively, the computing device may be formed as a server or a cloud-based system. A local or technical connection to the microscope is not mandatory; rather, it is sufficient if the computing device can load microscope images from data storage.

[0039] In conventional use, the properties of the invention described as additional microscope features also indicate variations of the method according to the invention. Conversely, the microscope system can also be configured to perform the described variations of the method. Variations of the computer program of the invention are given by the described variations of the method. Attached Figure Description

[0040] Other advantages and features of the invention are described below with reference to the illustrated drawings:

[0041] Figure 1 This is a schematic diagram illustrating a method for classifying interchangeable parts of a microscope based on existing technology;

[0042] Figure 2 This is a schematic diagram of an embodiment of the method of the present invention;

[0043] Figure 3 It is training Figure 2 A diagram illustrating the classification of methods and machine learning models.

[0044] Figure 4 This is a schematic diagram illustrating the training of a machine learning model for computing image processing according to an embodiment of the present invention; and

[0045] Figure 5 This is a schematic diagram of an embodiment of the microscope system of the present invention. Detailed Implementation

[0046] Various embodiments are described below with reference to the accompanying drawings. Identical and functionally equivalent components are generally identified by the same reference numerals.

[0047] Figure 2

[0048] Figure 2 An embodiment of the method according to the present invention using segmentation CNN 25 and classification CNN 40 is illustrated schematically.

[0049] First, a microscope image 20 is obtained, which at least shows a portion of the interchangeable part 9 of the microscope. The microscope image 20 can be loaded from memory or directly acquired and forwarded by the microscope. Here, the interchangeable part 9 is the cage 10, and is designed for… Figure 1 The given description of microscope image 20 and interchangeable parts 9 / cage 10 also applies here.

[0050] However, unlike Figure 1 Instead of feeding the classification CNN 40, the microscope image 20 is fed to the segmentation CNN 25, which computes a segmentation mask 31 as the processed image 30.

[0051] The segmentation CNN 25 is trained to output a binary mask, where one pixel value indicates that the corresponding pixel represents the interchangeable part 9, and the other pixel value indicates that the corresponding pixel does not represent the interchangeable part 9. Therefore, the label 11, screw 12, scratch 13, or other objects or textures of the interchangeable part 9 are only included in the microscope image 20, and not in the segmentation mask 31.

[0052] If segmentation mask 31 is now fed into classification CNN 40, it will not determine the category of interchangeable part 9 based on the label 11, screw 12, scratch 13, or other objects on the interchangeable part 9, but rather based on the shape of the interchangeable part 9 in segmentation mask 31. Specifically, the classification in this example uses the features shown in images 32 and 33. This includes the sample recess 14; that is, the size and location of the circular recess in segmentation mask 31 are used to distinguish different taught interchangeable part categories. As shown in image 33, the curved contour 15 of the outer periphery of the interchangeable part 9 is used as another distinguishing feature. This utilizes the knowledge that the shape of the interchangeable part 9 varies little or not at all within the interchangeable part category, while surface features on the interchangeable part 9 or smaller objects such as screws vary significantly more within the same interchangeable part category. For example, the geometry of the sample recess 14 within the same category of cage 10 hardly fluctuates, as different geometries have a stronger impact on functionality.

[0053] The classification CNN 40 ultimately produces output 45, which indicates the classification of segmentation mask 31 or the interchangeable component 9 shown therein. In the example shown, output 45 indicates which category of cage 10 was obtained.

[0054] As an advantage, unlike classic classification image processing algorithms, the texture features of the cage 10 have no effect on classification. Typical changes made to the cage 10 by the manufacturer and slight disturbances or variations in the microscope image 20 are significant obstacles for classic classification image processing algorithms, while in this case, the segmentation mask 31 remains essentially unchanged and therefore does not affect classification. Furthermore, when the number of categories is large or when new categories are added, classic classification image processing algorithms require time-consuming programming work, which in this embodiment can be easily accomplished through a new training process with additional training images.

[0055] In a variation of the illustrated embodiment, the segmentation CNN 25 is replaced by another neural network that computes image-to-image imaging. Therefore, the description of the segmentation mask 31 should be understood similarly for the processed image 30. For example, instead of the segmentation CNN 25, an autoencoder trained to compute a processed image 30 that is as consistent as possible with the input microscope image 20. The level of detail in this image-to-image imaging is essentially determined by the layer in the neural network with the fewest neurons. This corresponds to... Figure 2Bottleneck layer 26. By choosing a sufficiently small size for bottleneck layer 26, image details such as smaller labels 11, screws 12, and scratches 13 are not transferred to processed image 30. Unlike segmentation mask 31, the pixel values ​​of processed image 30 generated by autoencoder can have more different values; however, in both cases, processed image 30 does not have the aforementioned interfering image details.

[0056] In other variations, instead of the segmentation CNN 25, classical image processing algorithms can be used to compute the processed image 30 from the microscope image 20 without a machine learning model. The segmentation CNN 25 and the classification CNN 40 can also be connected to form a single neural network.

[0057] In variations of the described embodiments, the classification CNN 40 may be replaced by a separately designed neural network for classification or by a classic classification algorithm that does not use a machine learning model. The classification algorithm may, for example, be designed to evaluate the shape and / or size of the periphery and the possible openings of the interchangeable parts 9 shown in the segmentation mask 31. Regarding the example shown, the classification algorithm may, for example, be designed to distinguish different interchangeable part categories by referencing the size of the sample recess 14 and / or the distance from the sample recess 14 to the periphery of the interchangeable parts in the segmentation mask 31.

[0058] The described cage 10 should be understood only as an example of the replaceable part 9. Other microscope parts, especially those usually listed in the instruction manual, may also be interchangeable parts 9.

[0059] Figure 3

[0060] Figure 3 The training process of classification CNN 40 is shown in a simplified manner.

[0061] Using training image 31', which corresponds to Figure 2 The processed image 30. The training image 31' shows an interchangeable part, wherein the categories of the interchangeable parts shown are known and included in the training as the target variable T1.

[0062] The parameter values ​​of the classification CNN 40 should be learned or established during training. To this end, the classification CNN 40, with its initial parameter values, first computes the corresponding output 45 for each training image 31'. The machine learning application M1 feeds these outputs 45 to a loss function L1, which determines the difference from the target variable T1. Based on the result of the loss function L1, the machine learning application M1 modifies the parameter values. The above steps are repeated until the loss function is minimized or the output 45 is sufficiently consistent with the target variable T1. Optionally, association information K for the corresponding training image 31' can also be pre-given. The association information K may, for example, relate to microscope settings and be considered for learning classification. For example, the illumination settings may indicate which filter inserts or DIC slides are feasible or infeasible as interchangeable components. During continuous operation, the association information of the microscope images can also be stored together as microscope images or received by the microscope via a data interface.

[0063] The choice of training image 31' affects the features learned by CNN 40 for differentiation. In the example shown, a segmentation mask is used as training image 31'. Alternatively, a simulated segmentation mask can also be used as training image 31'. These can be generated, for example, by a CAD computer program that knows 3D data of interchangeable part types. Advantageously, the segmentation mask from simulated data and the segmentation mask from actual microscope images are substantially identical. Therefore, simulated segmentation masks can be implemented relatively easily to cover the entire range of variations appearing in a category. Furthermore, segmentation masks require fewer training images because the segmentation masks to be classified are less complex than microscope images. While the classification CNN 40 may structurally correspond to known neural networks, it may differ from these known neural networks in the training images used, thus differing in the learned parameter values.

[0064] Figure 4

[0065] Figure 4 The training process of CNN 25' is shown in a simplified manner, which can replace segmentation CNN 25 for computing processed image 30 from microscope image 20.

[0066] During training, training image 20' is used, which corresponds to the image from... Figure 2The training image 20' is used, and at least one interchangeable part is shown in each of the microscope images. Additionally, a target image T2 is used, corresponding to the training image 20', but differing in that surface features of the interchangeable parts are removed or less detailed in the target image T2, and / or objects (e.g., screws) on the interchangeable parts are removed in the target image T2. Using this training data, the CNN 25' learns how to process an input image / microscope image into an output image / processed image, where the shape of the interchangeable parts is received from the input image and is within the output image, and surface features and / or objects on the interchangeable parts have been removed from the input image. The training steps can be based on... Figure 3 The implementation is carried out, in particular, by comparing the computer program or machine learning application M2 with one or more loss functions L2, comparing the current output image / processed image 30 of CNN 25' with the target image T2', and changing the relevant parameter values ​​of CNN 25'.

[0067] Figure 5

[0068] Figure 5 An embodiment of a microscope system 100 according to the present invention is shown. The microscope system includes a computing device 50 and a microscope 1. In the illustrated example, the microscope is an optical microscope, but in principle, it could also be other types of microscopes.

[0069] Microscope 1 includes a support 5 via which other microscope components can be held. This support may include, in particular: an objective changer or turret, on which, in the illustrated example, objective 2 is mounted; a sample stage with a holder 10 for holding sample carrier 3; an aperture or filter insert 4 via which optical elements can be brought into the optical path; a microscope camera 6, which receives detection light via the selected objective 2; and a survey camera 7 for imaging a general overview image. The survey camera 7 is directly aimed at sample carrier 3, but sample carrier 3 can also be observed through a deflector. The deflector may, for example, be held on the objective changer and selected in place of objective 2.

[0070] The field of view 8 of the overview camera 7 is marked and covers the area around the location where the sample is to be positioned. Depending on the arrangement and design of the overview camera 7, various microscope components may be located within the field of view 8. Interchangeable or position-replaceable or adjustable microscope components are referred to herein as interchangeable components. In the illustrated example, the objective lens 2, sample carrier 3, holder 10, and / or aperture / filter insert 4 may be visible as interchangeable components in the overview image. In the sense of the present invention, the sample itself may also optionally be considered as an interchangeable component.

[0071] An overview image from the overview camera 7, or an image calculated therefrom, can represent the aforementioned microscope image. However, in principle, an image captured by the microscope camera 6, particularly via the microscope objective lens 2, can also form the aforementioned microscope image.

[0072] The acquired microscope images are transmitted to the computing device 50. Control of the microscope 1 can also be performed via the computing device 50 or alternatively via other electronic devices not shown. The computing device 50 can be arranged next to the microscope 1, spatially away from it, or in principle, it can also be arranged in the support 5 or other components of the microscope 1. Therefore, the computing device 50 can also be optionally considered as part of the microscope 1. The computing device 50 is configured to implement the method according to the invention, particularly calculating and processing images from microscope images, and then performing classification as described above.

[0073] An embodiment of the computer program according to the present invention includes instructions, the execution of which is referenced. Figure 2 The described steps and optionally the implementation reference Figure 3 The steps described in and / or 4. The computing device 50 may include a computer program.

[0074] The described embodiments are purely illustrative and variations thereof may be made within the scope of the appended claims.

[0075] List of reference numerals

[0076] 1. Microscope

[0077] 2 (Microscope) Objectives

[0078] 3 Sample carriers

[0079] 4. Aperture or filter insert

[0080] 5 supports

[0081] 6 Microscope Cameras

[0082] 7 Overview Camera

[0083] 8. Overview of the camera's field of view 7

[0084] 9 interchangeable parts

[0085] 10 cages

[0086] 11. Labels on interchangeable parts 9

[0087] 12 Screws or other objects on interchangeable parts 9

[0088] Scratches on interchangeable parts 9 (13)

[0089] 14 Sample recess on interchangeable part 9

[0090] The curved profile of the outer shape on the interchangeable part 9 of part 15

[0091] 20 microscope images

[0092] 20 training images / microscope images

[0093] Partial images of microscope images 21, 22, 23 (image portion 20)

[0094] 25 machine learning models, segmentation CNN

[0095] 25' Machine learning model or CNN for computational image processing

[0096] 26 Bottleneck Layers

[0097] 30 Image Processing

[0098] 30' Output Image / Process Image

[0099] 31-segment mask

[0100] Training images / training data for a 31' classification CNN

[0101] 32,33 Processing image portion of image 30

[0102] 40 categories of CNN

[0103] 40' Classification of Existing Technologies CNN

[0104] 45-class CNN40 output

[0105] Output of CNN40' (45' Classification)

[0106] 50 computing devices

[0107] 100 Microscope System

[0108] K-related information

[0109] L1, L2 loss functions

[0110] M1, M2 machine learning applications

[0111] T1 is the target variable 31' of the training images.

[0112] T2 is related to the target image 20' of the training image.

Claims

1. A method of classifying an interchangeable component (9) of a microscope, wherein the method comprises: obtaining a microscope image (20) in which the interchangeable component (9) is at least partially imaged; computing a processed image (30) from the microscope image (20); and performing an image evaluation on the processed image (30) to classify the interchangeable component (9); characterized in that the computation of the processed image (30) comprises removing image content of an object (11, 12, 13) at least in regions of the microscope image (20) that are converted into the processed image (30).

2. The method according to claim 1, wherein the removed image content is a surface feature.

3. The method according to claim 1, wherein the computation of the processed image (30) is performed by computing a segmentation mask (31) from the microscope image (20), wherein the segmentation mask (31) indicates which image regions represent the interchangeable component (9).

4. The method according to claim 1, wherein, the computation of the processed image (30) comprises filling image regions of the object (11, 12, 13) based on neighboring image content or cropping the image regions.

5. The method according to claim 1, wherein the microscope image (20) is fed to an autoencoder that computes the processed image (30) from the microscope image (20).

6. The method according to claim 1, wherein by removing the image content, the processed image (30) only shows the shape of the interchangeable component (9).

7. The method according to claim 1, wherein the computation of the processed image (30) is implemented by a trained machine learning model (25, 25’).

8. The method according to claim 1, wherein the object (11, 12, 13) from which image content is removed for computing the processed image (30) comprises one or more of: a label, a sticker, a scratch (13), a clip, a screw (12), and a smudge.

9. The method according to claim 8, wherein the label comprises a tag (11).

10. The method according to claim 1, wherein the interchangeable component (9) is a holder (10) of a sample carrier (3), a sample stage, an objective (2), an illumination module, a filter or prism, a condenser, a filter insert (4), a DIC slide, an assembly for polarization contrast recording, a light modulator, a sample carrier (3), or a sample.

11. The method according to claim 10, wherein the light modulator comprises a grating.

12. The method according to claim 1, wherein when performing the image evaluation on the processed image (30) for classifying the interchangeable component (9), the shape of the interchangeable component (9) is considered instead of the texture of the interchangeable component (9).

13. The method according to claim 1, wherein the classification of the interchangeable component (9) is implemented by a trained classification machine learning model (40).

14. The method according to claim 13, wherein a segmentation mask is simulated using a simulation program and the segmentation mask is used as training data (31') for the classification machine learning model (40).

15. The method according to claim 1, wherein the interchangeable component (9) is further classified using context information, the context information comprising a microscope setting or a sample property.

16. The method according to claim 15, wherein the sample property comprises a sample type.

17. The method according to claim 1, wherein a single machine learning model not only calculates the processed image (30) from the microscope image (20), but also image evaluates the processed image (30) to classify the interchangeable component (9).

18. A computer program product having instructions which, when executed by a computer, cause performing the method according to any one of claims 1-17.

19. A microscope system, having a microscope (1) and a computing device (50); wherein the microscope (1) comprises at least one interchangeable component (9) and is arranged for taking microscope images (20) in which the interchangeable component (9) is at least partially imaged; and wherein the computing device (50) is arranged for calculating a processed image (30) from the microscope image (20) and evaluating the processed image (30) to classify the interchangeable component (9); characterized in that the computing device (50) for calculating the processed image (30) is further arranged for removing image content of an object (11, 12, 13) at least in the area of the microscope image (20) which is converted into the processed image (30).

20. The microscope system according to claim 19, wherein the removed image content is a surface feature.

Images