Microscopy system, method and computer program for processing a microscopy image

Abstract

A method for processing a microscope image (10) comprising: obtaining (S1) a microscope image (10) taken by a microscope (1); identifying (S4) at least one image portion (11) having sensitive information within the microscope image (10) by means of an image processing program (B) using given reference information (i) about sensitive information; concealing (S6) the at least one identified image portion (11) having sensitive information to generate an anonymized image (20); and outputting (S7) the anonymized image (20). Further a corresponding microscope system is described.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a microscopy system and a method and a computer program for processing a microscopy image. BACKGROUND

[0002] In modern microscopes, digital image processing plays an increasingly important role. In particular, machine learning procedures are used more and more for numerous applications. For example, the applicant has developed methods based on machine learning procedures to perform automatic focusing, automatic sample container recognition, e.g. for documentation purposes or for navigation, measurement or distance determination of sample containers, or stitching of multiple taken images. The quality of these steps depends on the training data available to the machine learning procedure, etc. Therefore, it is desirable that the microscopy images available for use in the training cover a broad range of possible application fields. However, the transfer of microscopy images from different sources is often not possible, e.g. in the medical field when the microscopy images contain information related to a patient, or in research and development environments where the microscopy images indicate research results that should not be transferred to third parties. Therefore, for modern machine learning procedures for processing microscopy images, one challenge is the availability of comprehensive training data.

[0003] However, the present disclosure is not limited to these as exemplarily cited fields and generally relates to image processing of microscopy images containing sensitive information without forcing a reference to machine learning applications. SUMMARY

[0004] It can be seen as an object of the present invention to provide a method, a computer program and a microscopy system which make microscopy images with sensitive information more generally available by image processing in an efficient way.

[0005] This object is achieved by the method having the features of claim 1, by the computer program of claim 24 and by the microscopy system having the features of claim 25.

[0006] In the method for processing a microscopy image according to the invention, a microscopy image taken by a microscope is first obtained. At least one image portion with sensitive information within the microscopy image is identified by means of an image processing procedure using given reference information about the sensitive information. Then the at least one identified image portion with sensitive information is hidden to generate an anonymized image. The anonymized image is then output.

[0007] The computer program according to the invention comprises instructions which, when the computer program is executed by a computer, cause the execution of the method according to the invention.

[0008] The microscopic system according to the invention includes at least one microscope for capturing microscope images and at least one computing device. The computing device is configured to identify at least one image portion with sensitive information within the microscope image by means of an image processing program using given reference information about the sensitive information. The computing device is further configured to hide the identified at least one image portion with sensitive information to generate an anonymous image and output the anonymous image.

[0009] Anonymous images can be generated by designing image processing programs to independently locate image regions containing sensitive information and then removing this information from the microscope image. In these anonymized images, the image content, except for the sensitive information, is essentially preserved. This remaining image content can then be used for further processing or evaluation steps.

[0010] To illustrate this more clearly, consider a microscope image captured by a panoramic camera that includes a sample carrier with multiple sample chambers and textual labels containing information about the sample. This panoramic image should be part of the training data for a machine learning application that automatically locates the sample chambers within the panoramic image. For the machine learning application, the textual labels are not required, but their information about the sample is prevented from being passed on to the captured image, for example, to the developers of the machine learning application. A remedy is provided by locating the image area containing the textual labels and hiding that area, for example, by replacing the text with blank areas of the label (image restoration). By removing this sensitive information, the remaining image content of the microscope image can be freely used.

[0011] Optional design

[0012] Advantageous variations of the microscope system and method according to the invention are the subject of the dependent claims and are explained in the following description.

[0013] Microscopic images can be panoramic images or individual sample images. Panoramic images show at least a portion of the sample carrier, such as at least one edge of the sample chamber. In particular, panoramic cameras can be used, which may be additionally present in addition to a microscope camera. The panoramic camera can optionally be tilted to observe the sample carrier or stage, i.e., at an angle to the optical axis, which may correspond to the longitudinal axis of the currently active microscope objective. Alternatively, panoramic images can be captured using both a microscope objective and a microscope camera, in which case the difference between the panoramic image and the individual sample image is that a microscope objective with a smaller imaging scale is used for the panoramic image compared to the one used to capture the individual sample image. For some measurements, only the sample carrier may be visible in the panoramic image, while the sample itself may not be visible.

[0014] Identifying image portions with sensitive information

[0015] What is sensitive information can vary depending on the application case. The image processing program comprises reference information by which certain objects or image regions can be found in the microscope image. Thus, the reference information determines which image portions can be identified and sensitive information can be defined as those objects or image regions which can be located by the current reference information of the image processing program.

[0016] The pre-given reference information can in particular relate to one or more of the following: a coverslip region of the sample carrier; one or more sample chamber regions; one or more sample regions on the sample carrier and / or a background region outside the sample carrier. These types of sensitive information can in particular occur in the case of a panorama image as microscope image. The image processing program can locate in these cases where in the microscope image an image portion is located which is or comprises a coverslip region, a sample chamber region, a sample region on the sample carrier or a background region outside the sample carrier. There can be reference information here about different types of sample chamber regions, for example about angular and circular sample chambers and / or differently sized unit cells of various types of microtiter plates. The background region can for example show microscope components or a laboratory environment which the microscope user can not want to show, so the background region can generally be considered sensitive information.

[0017] The reference information can additionally or alternatively also relate to text annotations, labels or markings on the sample carrier. Thereby one or more image portions can be identified in the microscope image in which the sensitive information is a text annotation, label or marking on the sample carrier. The reference information can indicate that not every type of text annotation is evaluated as sensitive information. For example, letters and numbers which are associated with the sample carrier and which are not themselves a description of the sample taken or the experiment carried out, for example in the case of column and row numbers of a microtiter plate or the manufacturer name of the sample carrier, can be evaluated as non-sensitive text annotations. The text annotations which are classified as sensitive information can be handwritten or can be machine-generated and they differ in size, color and font. There can be reference information in order to always identify handwritten text as sensitive information.

[0018] Alternatively or additionally, the pre-given reference information about sensitive information can also relate to one or more sample types, in particular biological cells, organelles, tissue sections or pathological data. Thus, with this reference information, the image processing program can locate image portions that contain, for example, these specific organelles, pathological image regions that can be indicative of a specific patient or (textual) information that is relevant to the patient. If the object / sample to be examined is an electronic component, the reference information can also relate to the product identification or manufacturer name of the component.

[0019] The shape of at least one image portion with sensitive information can also be pre-given, that is, in particular independently of the current microscope image. The shape of the image portion is thus separate or independent from the shape associated with the sensitive information. For example, for this image portion, a rectangle or a circle can be pre-given as a shape, while the sensitive information relates to a biological cell type and the shape associated with the sensitive information is accordingly the shape of this biological cell. In other words, the image portion that is hidden can always include the environment around the sensitive information in one piece to mask the shape of the sensitive information.

[0020] The image processing program can be designed to offer the user a selection option as to which types of sensitive information the respective image portions are to be hidden for. In this case, the image processing program can use reference information for various types of sensitive information. The recognition and hiding then only relate to image portions for those types of sensitive information that have been selected by the user. For example, the user can select whether sample regions and / or text and / or the background are to be considered sensitive information.

[0021] However, as an alternative to manual selection, the image processing program can also be designed to select itself as to which types of sensitive information the respective image portions are to be hidden for. This selection is then made on the basis of the image content of the microscope image or context information for the microscope image. The image processing program can make the selection of the type of sensitive information in particular depending on one or more of the following factors:

[0022] The selection can be made depending on whether the microscope image is a panoramic image or a sample image. In the case of a panoramic image, for example, text annotations and / or background regions can be sensitive information, in contrast to sample images, in which these information are usually not contained. Conversely, certain cells as sensitive information are usually only found in sample images, but not in panoramic images.

[0023] The selection can also be made depending on parameters and settings on the microscope stand or microscope assembly, for example depending on the settings of the illumination, the filter, the camera settings or the microscopic method, for example phase contrast microscopy or fluorescence microscopy. Such settings can infer the type of experiment and thus the type of sensitive information specific to a certain sample type.

[0024] Furthermore, the selection can be made depending on the file name of the microscope image or information associated with the file of the microscope image. Such information can relate to file metadata, date of taking, creator, folder name and / or image resolution, for example.

[0025] Furthermore, the selection can be made depending on the sample type or sample carrier type used when taking the microscope image. For example, reference information about the position of the text description area can be associated with the sample carrier type as sensitive information.

[0026] Furthermore, the selection can also be made depending on the text content on the sample carrier. For example, it can be provided that text descriptions that are not specific to the experiment are not anonymized, in particular manufacturer and model names or column and row numbers of a multi-well plate, while other information, in particular about the sample under examination, is hidden.

[0027] Identifying image portions by means of a machine learning algorithm

[0028] The image processing program can comprise a trained reference model of a machine learning algorithm / machine learning model, which performs the identification of the at least one image portion with sensitive information. Reference information about the sensitive information is contained in the trained reference model here as model parameters, in particular model weights. The reference model can be a convolutional neural network (CNN), for example, which has a plurality of parameters determined in a training process. These parameters can represent weights or model weights.

[0029] The image processing program or the reference model can be designed to identify the at least one image portion with sensitive information by segmentation or detection. In the case of segmentation, the microscope image is divided into different regions, i.e. each image pixel is assigned to a cell. The image processing program can in particular be designed to identify various objects, some of which are considered sensitive information and others are not. For example, segmentation can determine the edge of the sample carrier and the sample chamber located therein, wherein the sample carrier area outside the sample chamber is considered non-sensitive information and the sample chamber containing the sample to be examined is considered an image portion with sensitive information.

[0030] The image processing program or the reference model can also be designed for classifying the microscope image, whereby the microscope image is assigned to one of a plurality of possible image classes. The respective positions of the at least one image portion with sensitive information are stored as reference information for the different image classes. The recognition of the image portion with sensitive information is now carried out by using the stored position of the determined image class. For example, the classification of the microscope image can reveal which type of sample carrier was used. For the various different sample carriers, it is known at which points text information fields are located, in which the detailed information of the individual, i.e. the sensitive information, is entered. Since the positions of these text information fields in the microscope image or relative to the sample carrier in the microscope image are known, the respective areas in the microscope image can be evaluated as image portions with sensitive information. In this case, the image portions are thus not evaluated as image portions with sensitive information on the basis of their content, but on the basis of their position in consideration of the reference information stored for the determined image class. These steps can be implemented with the aid of a machine learning algorithm or by a classic image processing program without machine learning algorithm.

[0031] The method of the application can also optionally comprise a training of the reference model of the machine learning algorithm in order to determine the model parameters. The training is carried out by pairs each consisting of one training image and one associated target image, wherein the training image contains image portions with sensitive information and in the associated target image the regions corresponding to these image portions are locally marked or changed, in particular hidden. By this training, for example, text information fields can be precisely classified as sensitive or non-sensitive information, either according to their text content or according to their position relative to the sample carrier.

[0032] The reference model of the machine learning algorithm can also be designed for carrying out the hiding. The microscope image can here be the input of the reference model, wherein its output is an anonymized image. For example, the image processing program can comprise a neural network, in particular a convolutional neural network CNN, trained with the aid of a machine learning algorithm, or a generative adversarial network GAN, which directly generates an anonymized image from a microscope image. For this purpose, the neural network is trained by pairs each consisting of one training image and an associated target image, wherein the training image contains image portions with sensitive information and in the associated target image the regions corresponding to these image portions are locally hidden.

[0033] Optionally, the image processing program can comprise an autoencoder trained by means of a machine learning algorithm, which directly generates the anonymized image from the microscope image. For this, the autoencoder is trained with training images which do not contain image portions with sensitive information. In general, an autoencoder can be understood as a functional pair consisting of an encoder and a decoder, wherein the encoder generates a low-dimensional encoding from an input image, and the decoder in turn generates an output image from this result of the encoder. Here, the encoder and the decoder are trained such that the error (deviation) between the output image and the input image is minimized. Thus, the autoencoder generates an output image which is usually very similar to the input image. This applies, for example, to data compression or simple visualization. Since the training data now only contains microscope images without sensitive information, the autoencoder is trained to be able to image only such input images without sensitive information exactly. If the input image deviates significantly from the training data, the autoencoder cannot or only very poorly describe this deviation, and the content of this deviation is missing in the output image of the autoencoder or is falsified in the output image. Since the training data does not contain any image portions with sensitive information, the autoencoder cannot transfer this information into the output image. In the sense of the present disclosure, the deletion of sensitive information by the autoencoder can be seen as the identification and concealment of image portions with sensitive information.

[0034] Concealing the identified image portion

[0035] Concealing the identified image portion with sensitive information It is to be understood that the content of the image portion is changed in such a way that the sensitive information can no longer be identified or is completely deleted. In this case, either the entire image portion (e.g. a rectangle) can be hidden, or only the part of the image portion that contains the sensitive information, e.g. the image pixels representing the letters of the sensitive information. The concealment can be carried out, for example, by pixelating the identified image portion, making noise, smoothing or blurring. Pixelation is understood as a number of adjacent image pixels obtaining the same value, resulting in a partial loss of image information. Noising characterizes a random change in the pixel values in the image portion, so that the original pixel values can no longer be reconstructed. Blurring, e.g. by a Gaussian filter, characterizes an averaging or smearing of a number of adjacent image pixels. Other image filters can also be used which change the image content to such an extent that the original content can no longer be identified or reconstructed, e.g. by forming an average value of the image pixels.

[0036] Alternatively, the hiding can also be carried out by replacing the recognized image portion or some parts of the image portion based on the content of the environment image. For example, the image pixels of the text description can be replaced or filled in accordance with the adjacent pixel values, which for example represent the text description area or the background. By such a filling, it is possible to continue the color or brightness gradient of the adjacent image area in the image portion to be hidden in particular. A particularly natural image impression can thereby be produced.

[0037] If the sensitive information is text, the hiding can also be carried out by replacing the text with another text, in particular a text without content. The new text can be adapted in size and font to the original text.

[0038] The hiding can also be carried out by means of a machine learning program. Thus, the recognized image portion can be fed to a neural network which is trained for filling in image areas by means of context information from adjacent areas. The output of the neural network then replaces or manipulates the recognized image portion. This process is also called inpainting. The neural network can in particular be designed as an image-to-image CNN or GAN.

[0039] Alternatively, the hiding can be carried out by means of an autoencoder. The recognized image portion is fed to an autoencoder which thereby generates an output which replaces the recognized image portion. The autoencoder is trained by means of training images which do not contain any sensitive information therein. Similar to the variant described, in which the autoencoder is used both for recognition and for hiding, the selection of the training data determines which information the autoencoder can reproduce and which information is missing in the output of the autoencoder. For example, the autoencoder is adapted for hiding a particular sample, wherein such a sample does not occur in the training data.

[0040] Alternatively, a particularly secure type of hiding can be achieved by cutting out or cutting off the recognized image portion. Thereby, no longer any image information is available for the image portion. The same applies when the recognized image portion is replaced by a pre-given content, i.e. independent of the content of the microscope image, for example by a pre-given pattern or a pre-given color.

[0041] A further possibility for hiding consists in deleting specific color channels of the microscope image. The microscope image can comprise a plurality of color channels, wherein each image pixel comprises a respective value per color channel. The individual color channels usually depict special cell structures, so that by deleting only one specific color channel, information about these cell structures is essentially deleted from the microscope image, while the information of the other color channels remains. Deleting one color channel can in particular mean that its values are set to a constant, for example zero. In this case, the deletion can take place only in the identified image portion, so that the image quality outside the identified image region is not impaired, or alternatively also for the entire microscope image, thereby preventing conclusions about the sample or sample properties from the used color channels.

[0042] The image processing program can optionally offer the user the choice of how the identified image is to be hidden. In this case, the image processing program can carry out the hiding in various different ways for this purpose. For example, the repair can be carried out in accordance with the standard, wherein the image pixels of the sensitive information are replaced by means of the surrounding pixels in order to achieve a realistic image impression. However, if the user considers that the sensitive information has not been or will not be sufficiently hidden, the user can for example select the replacement of the entire identified image portion with a pre-given pattern.

[0043] Optionally, the image processing program can be designed to carry out a type of hiding depending on the sensitive information. In this case, the various described methods of hiding can be selected, in particular depending on which of the above-mentioned sensitive information is found. In the case of an image portion with a textual specification as sensitive information, the described hiding can for example be carried out by means of replacement by the surrounding image content, while for an image portion with a non-textual specification as sensitive information, the described hiding is carried out in a different way than by means of replacement by the surrounding image content, for example by pixelation or by replacement by a pre-given content.

[0044] Using anonymous images

[0045] The application also relates to a method for using microscope images, wherein at least one computing device carries out the method according to what has been described in order to process the microscope images and to generate at least one anonymous image therefrom. The at least one computing device transmits the anonymous image to a central computing device which has no access to the taken microscope images. Thereby, the central computing device can essentially receive and use the information content of the microscope images except for the sensitive information.

[0046] For easier distinguishability, the terms "computing device" and "central computing device" are used, wherein both in principle can be or comprise any computer, server or cloud-based system. The computing device in particular can be connected with the microscope or assigned to a particular microscope and can also be used for controlling the microscope. For example, the computing device can comprise a personal computer or one or more processors, in particular as part of the microscope. Alternatively, the computing device can be formed by a server or cloud-based system. A local connection or technical connection to the microscope is not mandatory; rather, it is sufficient if the computing device can load the microscope images from a data storage.

[0047] The central computing device can comprise any computer, server or cloud-based computing system and is spatially separate from the microscope and the computing device. The central computing device can communicate with the computing device, for example, via the Internet or other networks. Alternatively, however, the data transmission between them can also be isolated without a fixed communication link. The terms "central computing device" and "computing device" are used for better linguistic differentiation, wherein there need not necessarily be a difference in physical design.

[0048] The at least one computing device can be set up to display the anonymized image to the user for approval and only transmit the anonymized image to the central computing device after approval. The user is thereby able to check whether sensitive information has actually been deleted. The computing device can also provide the user with an input tool with which further image sections can be manually marked for additional concealment. Information about which image sections have been manually marked can be transmitted together and used to improve the image processing program so that the image processing program is able to automatically identify all image sections with sensitive information as far as possible without exception.

[0049] Particular advantages arise when a plurality of computing devices each perform the described method for processing microscope images and thereby each generate at least one anonymized image and transmit it to the central computing device. In this case, the central computing unit also has no access to the photographed microscope images, but rather uses the anonymized images for joint data evaluation. The central computing device can therefore use the anonymized images, in particular in further processed form, as or for training images for machine learning applications. Thereby, the already mentioned example of a machine learning algorithm can be improved, or other machine learning algorithms not used for anonymized microscope images can be trained.

[0050] General properties

[0051] A microscope can in particular be understood as an optical microscope or also as a measuring device of different design, which is set up for taking an image (microscope image). According to a variant embodiment of the application, the image taking process can be part of the method or the method starts with loading an already existing microscope image.

[0052] As sample carrier, any carrier is understood, in which or on which a sample is to be accommodated. In particular, the sample carrier can be or comprise an object carrier consisting of e.g. glass, a multi-droplet plate, a culture dish, a capillary or a counting chamber. The sample carrier can comprise a plurality of sample chambers, which are understood as being laterally separated from each other areas for accommodating a plurality of samples.

[0053] An anonymized image is understood as an image derived from a microscope image, in which sensitive information contained in the microscope image has been removed or hidden. In this case, it is possible to hide the image parts containing the environment in addition to the sensitive information, or only the settings of the image parts showing the sensitive information are hidden. The remaining image area beyond the image parts can then remain unchanged or can also be changed according to the design. This is the case, for example, if color channels for the entire microscope image are removed or if an autoencoder or a GAN is used to calculate the area of the anonymized image from the area of the microscope image.

[0054] If the described process steps suggest an order, further processes can also be inserted therebetween. In particular, data processing and modifications can be made, for example, the microscope image does not have to be the original image taken by the microscope, but one or more such original images can first be processed to form the microscope image.

[0055] The computer program of the application can in particular comprise the described variants of the image processing program for generating an anonymized image. Furthermore, the computer program can comprise instructions by which the functions of the central computing device and the communication with the computing device are implemented. The computer program can consist of software packages which can be run independently of each other, wherein these software packages are to be executed on different computers, for example on the computing device and on the central computing device.

[0056] The properties of the application described as additional microscope features also lead to variants of the method according to the application when used as intended. In reverse, the microscopy system can also be set up to carry out the described method variants. BRIEF DESCRIPTION OF DRAWINGS

[0057] Further advantages and features of the application are described below with reference to the attached schematic drawings:

[0058] Figure 1A microscope image with sensitive information and an anonymized image calculated therefrom according to the application are schematically shown;

[0059] Figure 2 Another microscope image with sensitive information and an anonymized image calculated therefrom according to the application are schematically shown;

[0060] Figure 3 Another microscope image with sensitive information and an anonymized image calculated therefrom according to the application are schematically shown;

[0061] Figure 4 Another microscope image with sensitive information and an anonymized image calculated therefrom according to the application are schematically shown;

[0062] Figure 5 An embodiment of a microscopy system of the application is schematically shown;

[0063] Figure 6 A flow of an embodiment of a method of the application is schematically shown;

[0064] Figure 7 A flow of a machine learning algorithm of an embodiment of a method of the application is schematically shown;

[0065] Figure 8 A flow of a machine learning algorithm of an embodiment of a method of the application is schematically shown; and

[0066] Figure 9 An embodiment of a microscopy system of the application is schematically shown. DETAILED DESCRIPTION

[0067] Various embodiments are described hereinafter with reference to the drawings. Like and same parts are generally identified with the same reference numerals.

[0068] Figures 1 to 4

[0069] Figures 1 to 4 A microscope image 10 taken by a microscope is schematically shown in each case. In the case of the first image, a panoramic image is shown which shows a sample carrier 12. Depending on the measurement case, a sample 15 held by the sample carrier 12 can be seen in the panoramic image. Figures 1 to 3 A sample image taken at a higher magnification than the panoramic image is shown. In the sample image, the structure of the actual sample can be seen, here various biological cells. Figure 4 A sample image taken at a higher magnification than the panoramic image is shown. In the sample image, the structure of the actual sample can be seen, here various biological cells.

[0070] The transfer of the microscope image 10 facilitates various different applications. This is the case, for example, in the context of technical support or for the improvement of machine learning applications. In the context of technical support, the microscope user can often be given more targeted support if there is a microscope image 10 taken by the microscope user. For machine learning applications, there should be training data that cover the real-world measurement situation as well as possible; for this purpose, it is advantageous if microscope images 10 taken by microscope users can be used for the training data. However, this is often not possible because the taken microscope images 10 contain sensitive information that the microscope user is not allowed or does not want to transfer. Sensitive or critical information can be, for example, a text description, a sample region in a panoramic image or a special sample structure in a sample image.

[0071] The microscope image 10 from Figure 1 shows a sample carrier 12 with a cover glass 13, below which a sample 15 is located. The image region of the sample 15 is here classified as sensitive information that should not be transferred.

[0072] In a similar way, the microscope image 10 from Figure 2 shows an image region of the sample 15 that is classified as sensitive information and should not be transferred. In addition, the microscope image 10 also contains a text description area 14, which has, among other things, information about the sample 15. The text content of the text description area 14 is also classified as sensitive information that the microscope user is not allowed to transfer to third parties.

[0073] In the case of Figure 3 , the sample carrier 12 can be seen in the microscope image 10, on the cover glass of which a handwritten note has been made that is classified as sensitive information and should not be transferred. The particular details of the sample 15 cannot be seen here, so it will not be treated as sensitive information here.

[0074] The microscope image 10 from Figure 4 shows a part of the sample, here several cells, in which a cell or sample structure 16 specific to the experiment is located. For the microscope user, the special sample structure 16 represents sensitive information, while the other regions of the sample do not contain any information to be protected.

[0075] By means of the embodiments of the application, an anonymized image 20 is calculated from the microscope image 10, respectively. The image portion 11 of the microscope image 10 with sensitive information is replaced in the anonymized image 20 by a hidden region 21. Apart from the difference that the sensitive information has been removed, the anonymized image 20 thus essentially corresponds to the microscope image 10. In Figure 1In the example of Fig. 1, the cover glass region 13 is located in the microscope image 10 and this region is replaced by a pattern in the anonymized image 20. In Figure 2 In the example of Fig. 2, the image portion 11 is pixelated with sample 15 and thus cannot be distinguished. Furthermore, the text description region 14 is recognized as such and the text located thereon is cut off. Figure 3 The image pixels showing the text description are filled with environment image pixels, thereby resulting in a natural image impression. In Figure 4 The case of Fig. 3, an image region of a special sample structure 16 is hidden by blacking out the region 21. The anonymized image 20 can now be passed to e.g. technical customer support, a provider of image processing tools or a developer of a machine learning based application. Other exemplary use possibilities are storage on a location classified as insecure, e.g. on an external server, or use of a third party provider's cloud based image analysis service.

[0076] Figure 5 Embodiment

[0077] In Figure 5 One embodiment of a microscopy system 100 according to the present application is schematically shown in Fig. 4. The microscopy system 100 comprises a microscope 1 with a microscope stand 5, at least one microscope objective 2, a microscope camera 6 receiving detection light guided via the microscope objective 2, a light source for illuminating a sample and a light guide for guiding the light of the light source onto the sample to a condenser 4. The microscope 1 further comprises a sample stage 9 by which a sample carrier 12 can be held. The optical axis 3 of the microscope objective 2 is generally perpendicular to the surface of the sample stage 9 or sample carrier 12.

[0078] In the shown example, the microscope 1 comprises at least one additional panoramic camera 7, the field of view 8 of which is shown in dashed lines. The panoramic camera 7 is aimed at the sample carrier 12 in order to take a (macroscopic) panoramic image of the sample carrier 12. Thus, a larger portion of the sample carrier 12 is imaged in the panoramic image than in the sample image taken by the microscope objective 2 and following microscope camera 6. In one variant of the shown embodiment, the panoramic camera 7 does not directly view the sample carrier 12, but does so through a deflection mirror. The deflection mirror can for example be arranged on the nosepiece and selected to replace the microscope objective 2.

[0079] Instead of the shown upright microscope arrangement, a reversed arrangement can also be provided. The positioning of the panoramic camera 7 can also vary. Furthermore, instead of a separate panoramic camera 7, it can also be provided that the microscope camera 6 takes the panoramic image by switching to a microscope objective with a smaller imaging scale.

[0080] The microscope images described here can be panoramic images or sample images. They are sent to a computing device 40 which is set up to generate the anonymized images shown in Figure 1 The computing device 40 can be designed as part of the microscope 1 or as an additional unit, wherein the microscope 1 and the computing device 40 together form a microscopy system 100. The computing device can be composed of an electronic device arranged on the microscope 1 or in the microscope stand 5. Alternatively, the computing device 40 can be a computer arranged next to the microscope 1 or can be arranged remote from the microscope 1 and receive the microscope images from the microscope 1 or a data storage connected with the microscope 1 via a communication link.

[0081] The computing device 40 can be set up to execute an embodiment of the method according to the application as described with reference to the following figures.

[0082] Figure 6 Embodiments

[0083] An embodiment of a method for processing microscope images 10 according to the application is shown schematically in Figure 6 A computer program according to the application is designed to execute the method.

[0084] First, at least one microscope image 10 is obtained in step S1, for example from a memory or received from a microscope. In an optional step S2, various selection options il-i4 related to the sensitive information types are offered to a user. Thereby, the user can select which information should be classified as sensitive information and should be deleted from the microscope image. For example, the user can specify a text annotation il, a sample area i2 or a cover glass area i3 as sensitive information. Alternatively, the user can also specify an automatic selection i4.

[0085] Depending on the selection in step S2, the respective pre-given reference information i on the various sensitive information is used. The reference information i is stored in a memory or as data of the image processing program. The reference information i can include information for recognizing the respective structures in the image, for example information for character recognition. Additionally or alternatively, the reference information can also include position information which gives the position of the reference information relative to the sample carrier. For example, the position information can indicate where the text annotation area is located on the sample carrier. As reference information i, the current position information can also be a direct coordinate of the microscope image 10 and thereby define an image portion in the microscope image 10.

[0086] With the help of the reference information i, at least one image portion 11 having sensitive information is located in the microscope image 10 in step S4. The segmentation of the microscope image 10, which can be carried out for example by image processing, to divide the image into different regions, such as a “cover slip region”, a “sample carrier region outside the cover slip” and a “background outside the sample carrier” can be used for example. If it is determined what type of sample carrier it is, the reference information stored for this sample carrier about the location of the text field can be used. This information is used to locate the image region of the text field in the microscope image. By way of example, alternatively, all cover slip regions or all sample chambers with samples located therein can be identified as image portions having sensitive information.

[0087] Optionally, the context information determined for the microscope image in the previous step S3 can also be taken into account in step S4. The context information can be for example information associated with the file of the microscope image, such as a file name or folder name, or information about the measurement, such as the settings of the microscope. For example, from this information it can be inferred that a series of measurements was carried out using the same sample carrier and / or the same type of cover slip. From the context information it can also be inferred whether the microscope image is a sample image or a panorama image. Depending on the type of sensitive information, in particular a cover slip region, a label or a text field, the search is optionally carried out only in panorama images and not in sample images. A user name or a company name can also be used as context information, with which a selection of certain sensitive information to be hidden is associated, for example, for a certain user the cover slip region should always be hidden.

[0088] After the image portion having sensitive information 11 has been located in step S4, this image portion is hidden in step S6, for example blackened or replaced by a pre-given pattern. Thereby an anonymized image 20 is generated, which is finally output in step S7.

[0089] Optionally, a step S5 can be added, in which the user is offered selection options as to how the sensitive information should be made indiscernible. The selection options can for example include blackening, pixelization, blurring, filling with environmental content or replacing text with text without content. Step S5 can be carried out at principle any point before step S6 is carried out.

[0090] Figure 7 and Figure 8

[0091] for Figure 6 The described step S4 of identifying the image portion having sensitive information is carried out by an image processing program B. In a variant embodiment of the image processing program B, the image processing program B comprises a reference model R of a machine learning algorithm M. In a previous training process, the parameters of the model, the model weights, are determined. These model weights represent the reference information i. Figure 7 In a variant embodiment of the image processing program B, the image processing program B comprises a reference model R of a machine learning algorithm M. In a previous training process, the parameters of the model, the model weights, are determined. These model weights represent the reference information i.

[0092] Figure 7 A training process is shown, for which training images 30 are used, for which there are target images T. The target images T can for example correspond to the training images 30, in which respectively one image portion with sensitive information is marked. The target images T can be generated manually. The reference model R obtains as input each of the training images 30 and calculates therefrom an output. For this calculation, the reference model R uses parameters / model weights which do not yet have final values at the beginning of the training. The output of the reference model R is here the output images 32. The machine learning algorithm M uses a loss function L or alternatively a gain function which calculates the deviation between the current output image 32 and the associated target image T. Depending on this deviation, an optimization function of the machine learning algorithm M changes the current model weights of the reference model R. The above-described process is repeated until the loss function L is minimized or the alternative gain function is maximized. When the microscope image 10 is fed to the reference model R, the model weights determined at the end of the training can then be used.

[0093] In a variant of the described embodiment, the target images T are replaced by target data which are image coordinates or other information which specify one or more image portions of the respective associated training image 30. In a corresponding manner, the output of the reference model R is likewise image coordinates or other information by which the image portions are defined at this time.

[0094] The image portions determined by the reference model R are then concealed. This concealment can take place by means of classical image processing means or by means of a second machine learning algorithm shown in Figure 8 The second machine learning algorithm M2 comprises a second reference model R2 which obtains as input the output images 32 of the (first) machine learning algorithm M. In the associated target images T2 for the second machine learning algorithm M2, the image portions marked in the output images 32 are concealed, whereby the second machine learning algorithm M2 learns to calculate from the output images 32 as input an anonymous image 20 with concealed image portions.

[0095] Alternatively, the reference model R shown in Figure 7 The reference model R shown in

[0096] In Figure 7 andFigure 8 In particular, supervised learning by means of a deep learning model is used, for example by means of an image-to-image CNN (Convolutional Neural Network) or by means of a GAN (Generative Adversarial Networks). In certain designs, unsupervised or partially supervised learning can also be used.

[0097] Figure 9 Embodiments

[0098] Figure 9 Embodiments of a microscopy system 100 according to the application and embodiments of a method for utilizing microscope images 10 according to the application are shown. A plurality of microscopes 1 each take one or more microscope images 10 and provide them to a respective computing device 40. In the manner already described, the computing device 40 generates from the microscope images 10 an anonymous image 20. The computing device 40 then transmits the anonymous image 20 to a central computing device 50. The central computing device 50 is set up for jointly analyzing the anonymous images 20 from a plurality of computing devices 40 and thus from a plurality of microscopes 1. For example, the anonymous images 20 originating from different microscopes 1 can jointly serve as training data for a machine learning application 55. The training data can optionally also include further images 25. By means of this training data, the central computing device determines, for example, model parameters of an image processing application, which is then transmitted to the various computing devices 40. The computing devices 40 can now evaluate the microscope images 10 by means of the image processing application. In this case, all computing devices 40 benefit from the training data obtained from the information of the microscope images of the various computing devices, while the computing devices do not have to communicate microscope images with sensitive information.

[0099] By means of the various inventive refinements described, the relevant information of the microscope images can be used for many applications in which, otherwise, due to the sensitive information in the microscope images, the transmission and use of the data would not be possible or at least not desirable. Here, the microscope user does not have to manually remove information from the microscope images or the sample carriers. The described embodiments are purely illustrative and variations thereof are possible within the scope of the claims appended hereto.

[0100] List of reference signs

[0101] 1 microscope

[0102] 2 objective

[0103] 3 optical axis of the objective 2

[0104] 4 condenser

[0105] 5. Microscope stand

[0106] 6. Microscope camera

[0107] 7 Panoramic Camera

[0108] 8. Field of view of a panoramic camera

[0109] 9. Sample Stage

[0110] 10 Microscopic images

[0111] 11. Image portion in a microscope image

[0112] 12 Sample carrier

[0113] 13 Cover glass area

[0114] 14. Text Description Area

[0115] 15 samples

[0116] 16 Special Sample Structures

[0117] 20 Anonymous Images

[0118] 21 Hidden Areas

[0119] 25 images

[0120] 30 training images

[0121] 32 Output Image

[0122] 40 Computing devices

[0123] 50 Central computing devices

[0124] 55 Machine Learning Applications

[0125] 100 Microscopic System

[0126] B Image Processing Program

[0127] I. Reference Information Regarding Sensitive Information

[0128] i1-i4 Options for selecting the type of sensitive information

[0129] M Machine Learning Algorithm

[0130] M2 Second Machine Learning Algorithm

[0131] R reference model

[0132] R2 Second Reference Model

[0133] S1-S7 Method Steps

[0134] T, T2 target image

Claims

1. A method for processing a microscope image (10), comprising: obtaining (SI) a microscope image (10) taken by a microscope (1); identifying (S4) at least one image portion (11) having sensitive information within the microscope image (10) by means of an image processing program (B) using given reference information (i) about sensitive information; hiding (S6) the at least one identified image portion (11) having sensitive information to generate an anonymized image (20); and outputting (S7) the anonymized image (20); providing a selection option (S2) to a user, which kind of sensitive information respective image portions (11) are to be hidden for, wherein the given reference information (i) relates to different types of sensitive information, wherein the identifying and hiding only include image portions (11) for those types of sensitive information that have been selected by the user; or the image processing program (B) makes the selection of the type of sensitive information depending on one or more of the following factors: - depending on whether the microscope image (10) is a panorama image or a sample image; - depending on parameters and settings on a microscope stand (5) or microscope assembly; - depending on a file name of the microscope image (10) or information linked to the file of the microscope image (10); - depending on a sample type or sample carrier type used when taking the microscope image (10); - depending on textual content on a sample carrier (12).

2. The method according to claim 1, the microscope image (10) is a panorama image of a sample carrier (12) or a sample image of a microscopy sample (15). characterized in that 3. The method according to claim 1, the reference information (i) about sensitive information relates to one or more of the following: textual instructions, labels or markings on a sample carrier (12), cover glass area (13) of a sample carrier (12); one or more sample chamber areas; one or more sample areas on a sample carrier (12) and background areas outside the sample carrier (12); characterized in that wherein the microscope image (10) is a panorama image and wherein the at least one image portion (11) having sensitive information is a textual instruction, label, marking, cover glass area (13), one or more sample chamber areas, one or more sample areas on a sample carrier (12), or a background area outside the sample carrier (12).

4. The method according to claim 1, the pre-given reference information (i) about sensitive information relates to one or more sample types. characterized in that 5. The method according to claim 4, the pre-given reference information (i) about sensitive information relates to one or more sample types, the sample types being biological cells, organelles, tissue sections, pathological data or manufacturer or product names. characterized in that 6. The method according to claim 1, ​ characterized in that The image processing program (B) selects which types of sensitive information are to be hidden for the respective image portions (11) based on the image content of the microscope image (10) or on context information for the microscope image (10).

7. The method according to claim 1, characterized in that The shape for the at least one image portion (11) is predefined, whereby the shape of the at least one image portion (11) is separated from the shape associated with the sensitive information.

8. The method according to claim 1, characterized in that The image processing program (B) comprises a trained reference model (R) of a machine learning algorithm (M) which performs the recognition of the at least one image portion (11) with sensitive information, wherein the reference information (i) about sensitive information is contained as model parameters in the trained reference model (R).

9. The method according to claim 8, further comprising: The reference model (R) of the machine learning algorithm (M) is trained to determine model parameters, wherein the training is carried out by pairs each consisting of one training image (30) and one associated target image (T), wherein the training image (30) contains image portions (11) with sensitive information and in the associated target image (T) the regions corresponding to these image portions (11) are locally marked or hidden.

10. The method according to claim 1, characterized in that The image processing program (B) recognizes the at least one image portion (11) with sensitive information by means of segmentation or detection.

11. The method according to claim 1, characterized in that The image processing program (B) performs a classification of the microscope image (10), whereby the microscope image (10) is assigned to an image class, wherein the positions of the image portions (11) with sensitive information are stored as reference information (i) for different image classes, wherein the recognition of the at least one image portion (11) with sensitive information is carried out by using the stored positions for the determined image class.

12. The method according to claim 8, characterized in that The reference model (R) of the machine learning algorithm (M) also performs the hiding.

13. The method according to claim 1, characterized in that The image processing program (B) comprises a neural network trained by means of a machine learning algorithm (M), To this end, the neural network is trained by pairs each consisting of one training image and an associated target image, wherein the training image contains image portions (11) with sensitive information and in the associated target image the regions corresponding to these image portions (11) are locally hidden.

14. The method according to claim 13, characterized in that The machine learning algorithm (M) is a convolutional neural network CNN, or a generative adversarial network GAN, which directly generates the anonymous image (20) from the microscope image (10).

15. The method according to claim 1, characterized in that The image processing program (B) carries out the hiding by means of an autoencoder trained by means of a machine learning algorithm (M), by training the autoencoder with training images which do not contain image portions (11) with sensitive information therein, the autoencoder generating the anonymized image (20) directly from the microscope image (10).

16. The method according to claim 1, characterized in that The hiding is carried out by - pixelating, noise or blurring the identified image portion (11); or - replacing the identified image portion (11) or some parts of the identified image portion on the basis of the environmental image content; or - cutting off or cutting out the identified image portion (11); or - with a pre-given content; or - deleting the content of a specific color channel of the microscope image (10) at least in the identified image portion (11), 17. The method according to claim 1, The hiding is carried out by feeding the identified image portion (11) to a neural network which is trained for filling in image regions by means of context information from adjacent regions, and the output of the neural network replaces the identified image portion (11). characterized in that 18. The method according to claim 1, The hiding is carried out by feeding the identified image portion (11) to an autoencoder which generates an output replacing the identified image portion (11) therefrom, wherein the autoencoder is trained by means of training images which do not contain sensitive information. characterized in that 19. The method according to claim 1, A type of hiding is carried out in dependence on the sensitive information. characterized in that 20. A method for using a microscope image (10), at least one computing device (40) carries out the method according to any one of claims 1 to 19, characterized in that wherein the at least one computing device (40) transmits the anonymized image (20) to a central computing device (50) which has no access to the taken microscope image (10).

21. The method according to claim 20, The at least one computing device (40) displays the anonymized image (20) to a user for approval and transmits the anonymized image (20) to the central computing device (50) only after approval has been made. characterized in that 22. The method according to claim 20, A plurality of computing devices (40) each carry out the method according to any one of claims 1 to 19 and thereby each output at least one anonymized image (20), characterized in that wherein the plurality of computing devices (40) transmit the anonymized images (20) to the central computing device (50) which has no access to the taken microscope image (10), wherein the central computing device (50) carries out a common data evaluation of the anonymized images (20).

23. The method according to claim 22, The central computing device (50) uses the anonymized images (20) as training images for a machine learning application (55) and transmits the trained machine learning application (55) to the computing devices (40) after the training has ended. characterized in that ​ 24. Computer program having instructions which, when the computer program is executed by a computer, cause the method according to any one of claims 1-23 to be performed.

25. Microscopy system comprising: at least one microscope (1) for taking microscope images (10) and at least one computing device (40) which is set up for: - identifying, by means of an image processing program (B), at least one image portion (11) having sensitive information within the microscope images (10), the image processing program using given reference information (i) about sensitive information; - concealing the at least one identified image portion (11) having sensitive information to generate an anonymized image (20); and - outputting the anonymized image (20); the image processing program (B) comprising a neural network trained by means of a machine learning algorithm (M), To this end, the neural network is trained by pairs each consisting of a training image and an associated target image, wherein the training image contains image portions (11) having sensitive information and in the associated target image the regions corresponding to these image portions (11) are concealed locally.

26. Microscopy system according to claim 25, characterized in that the machine learning algorithm (M) is a convolutional neural network, CNN, or a generative adversarial network, GAN, which directly generates the anonymized image (20) from the microscopy image (10).

Images