Image-based maintenance prediction and malfunction detection

DE502019012297D1Active Publication Date: 2024-11-07CARL ZEISS MICROSCOPY GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
DE502019012297
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Priority Date
2018-12-20
Filing Date
2019-12-12
Publication Date
2024-11-07
Estimated Expiration
2039-12-12

AI Technical Summary

Technical Problem

Modern microscope systems require frequent and costly maintenance due to the need for additional sensors and continuous internet connections for predictive maintenance, which is not feasible for all setups, especially in sensitive environments like medical and materials science where downtime should be minimized.

Method used

A computer-implemented method using image analysis to determine the need for maintenance by evaluating digital images recorded during regular operation, eliminating the need for additional sensors and internet connections by leveraging existing image data for predictive maintenance.

Benefits of technology

This approach allows for predictive maintenance without additional sensors or internet connections, enabling precise detection of malfunctions and future errors, reducing downtime and maintenance costs by using existing image data from normal operation, thus improving the overall performance and longevity of microscope systems.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader
Need to check novelty before this filing date? Find Prior Art

Description

Field of the invention

[0001] The invention relates generally to a system for predicting the need for maintenance measures and, more particularly, to a computer-implemented method for predictive maintenance for a microscope system, a corresponding maintenance monitoring system and a computer program product. Technical background

[0002] Modern technical equipment often requires complex and time-consuming maintenance measures according to a predefined maintenance plan. This plan uses past experience to determine maintenance intervals so that the technical equipment does not fail due to lack of maintenance. This circumstance results in maintenance measures actually being performed more frequently than is necessary for some of the technical equipment. This is especially true when the equipment is handled with care and in a system-appropriate manner by high-quality personnel. This applies not only to pilots, who sometimes even receive increased compensation for handling engines with care (very rarely under full load), but equally to high-precision instruments such as highly complex precision microscope systems.These are used in fields such as medicine, biomedicine, or materials science; and here, too, system downtime due to maintenance work should be kept as short as possible in order to ensure the fastest possible return on investment in these devices.

[0003] Incorrect operation can lead to shortened maintenance cycles or even unexpected maintenance incidents. Modern maintenance approaches therefore take the actual use of a system into account. In rare cases, digital twins of the systems are already available – especially at the system manufacturers – to dynamically determine maintenance cycles. However, this would require a constant electronic connection (internet) between the microscope system and the manufacturer. For data protection and other reasons, such a connection is often undesirable. Furthermore, this type of continuous monitoring would require the microscope systems to be equipped with a multitude of additional sensors that enable precise recording of the handling and use of the microscope systems.However, equipping the system with additional sensors would entail significant additional costs to enable manufacturer-recommended dynamic maintenance cycles in the sense of predictive maintenance. The concept of predictive maintenance envisages that maintenance work is only performed when it is actually expected to be required. To enable this dynamic, predictive maintenance, it is typically necessary to obtain data on system usage, as would be provided by additional sensors within the microscope systems. As already outlined, this results in additional, not insignificant technical effort and significant additional costs.

[0004] A similar publication already exists: The document by MOHAMMAD QASHIDH ET AL: "Autonomous Scanning Probe Microscopy in-situ Tip Conditioning through Machine Learning", ARXMORG, March 19, 2018 (2018-03-19), XPO80865936, describes a scanning probe microscope that relies on an atomically sharp tip for scanning. A machine learning method is used to determine the condition of the probe tip.

[0005] The underlying task of the concept presented here can therefore be seen as enabling predictive maintenance without the need for complex additional internal sensors in the microscope system and / or additional network connections for the concept of a digital twin. Overview of the invention

[0006] This object is achieved by the computer-implemented method proposed here for detecting the need for predictive maintenance for an optical microscope system, a corresponding maintenance monitoring system for predictive maintenance for an optical microscope system, and a computer program product according to the independent claims. Further embodiments are described by the respective dependent claims.

[0007] According to one aspect of the present invention, a computer-implemented method for determining a need for predictive maintenance for an optical microscope system is presented. The method comprises providing an optical microscope system having an image sensor for capturing digital images of a sample, and capturing digital images using the image sensor during regular operation of the optical microscope system.

[0008] A captured digital image is provided to an image analysis system as at least one input data set. Based on this data, the image analysis system determines at least one feature in the at least one captured digital image, wherein the feature correlates with a malfunction of the optical microscope system during normal operation.

[0009] The method further comprises generating a status signal for a state of the optical microscope system by the image analysis system on the basis of the at least one digital image and the at least one feature contained therein, and determining - by using the status signal - a component group of the optical microscope system for which a maintenance measure is required.

[0010] According to another aspect of the present invention, a maintenance monitoring system for determining a need for predictive maintenance for an optical microscope system is presented. The maintenance monitoring system includes an optical microscope system having an image sensor for capturing digital images of a sample and an image sensor for capturing digital images during regular operation of the optical microscope system.

[0011] The maintenance monitoring system further comprises an image analysis system adapted to receive at least one recorded digital image as at least one input data set for an image analysis system. The image analysis system is adapted to determine at least one feature in the at least one recorded digital image by the image analysis system, wherein the feature correlates with a malfunction of the optical microscope system during regular operation. Furthermore, the maintenance monitoring system comprises generating a status signal for a state of the optical microscope system by the image analysis system based on the at least one digital image and the at least one feature contained therein, and determining, by using the status signal, a component group of the optical microscope system for which a maintenance measure is required.

[0012] Furthermore, embodiments may be implemented in the form of a corresponding computer program product accessible through a computer-usable or computer-readable medium comprising program code for use by or in connection with a computer or instruction execution system. For the context of this description, a computer-usable or computer-readable medium may be any device comprising means for storing, communicating, forwarding, or transporting the program for use by or in connection with an instruction execution system, corresponding device, or apparatus.

[0013] The computer-implemented method for determining predictive maintenance for an optical microscope system has several advantages and technical effects:

[0014] The proposed concept allows the implementation of the principle of predictive maintenance without consistently pursuing the Internet of Things (Industry 4.0), i.e., without equipping the microscope systems with additional, complex, internal sensors. The proposed concept does not require additional, cost-intensive recording and collection of temperature data, voltage values, and / or usage data directly from the components of the microscope system to be monitored.

[0015] Rather, the proposed concept is based on the evaluation and use of data already generated during normal operation and use of the microscope system. Furthermore, it is not necessary to record additional (reference) images. The analysis for determining the need for predictive maintenance is based on the images captured by a user during normal use of the device, for example, in experiments. Additional maintenance or calibration recordings are not required.

[0016] In addition, recording and evaluating the images recorded during normal operation in time series allows for a longer-term analysis of usage behavior, which in turn can be used to identify existing or potentially future errors in the microscope system.

[0017] A comparison between several microscope systems (possibly of the same series / type) can also contribute to an even more precise prediction of the need for maintenance.

[0018] The proposed concept allows for a status determination and detection of existing system errors - e.g. defective components, misalignment, etc. - an estimation of future errors - including a time horizon - and a root cause analysis - e.g. a differentiation between device-related, sample-related and usage-related errors.

[0019] Most of the disadvantages of previously known methods for improving the maintenance of microscope systems can be overcome by the concept proposed here: (a) Additional required sensors often cannot be retrofitted for technical or cost reasons; (b) deriving the device status from individual measurement parameter values ​​of individual sensors on individual components of the microscope system often does not allow for an overall assessment of the condition of the microscope system or requires a very complex set of rules and a large number of sensors and corresponding measured values. Nevertheless, these are often not representative for assessing the overall condition of a microscope system.

[0020] With the newly proposed concept, however, it is possible to compensate for user errors by adjusting the microscope system's acquisition parameters, thus exploiting the microscope system's full performance. Furthermore, the use of the microscope system by different operators, for whom the additional internal sensors would provide different measurement values, can also be compensated for when assessing the device's condition.

[0021] The use of artificial intelligence, image recognition, and machine learning techniques now enables the evaluation of the output of the microscope system, namely one or more acquired images during regular operation of the microscope system, in order to draw conclusions about the entire microscope system or its individual components based on their assessment and the "experience" from previously acquired images with known system errors using a trained neural network, as an example of an image analysis system.

[0022] No reference samples, reference patterns, or reference settings are required. Normal operation of the microscope system is sufficient to enable continuous assessment / evaluation of the overall condition of the microscope system, allowing for the identification, recommendation, and planning of preventive maintenance measures to minimize microscope system downtime and maximize user and / or customer satisfaction.

[0023] The main advantage is that necessary or recommended maintenance measures do not have to be supplemented or expanded with additional devices and / or extensions – in particular mechanical, electromechanical and / or electronic sensors – but that existing components of the microscope system are used to determine the need for predictive maintenance measures. This does not increase the complexity of the microscope system, so that no additional maintenance measures are required for these unnecessary extensions. Rather, the existing subsystems are utilized and – in accordance with the method – better utilized in the proper, routine, regular operation of the microscope system, in particular through improved evaluations of the acquired digital images, which are recorded anyway during regular operation. A correspondingly adapted image analysis system forms the basis for this solution.

[0024] In summary, it can be said that the secondary use of the recorded images - ie the evaluation of the images already acquired during regular operation of the microscope system with regard to necessary maintenance measures - enables a significant improvement in the utilization times of microscope systems.

[0025] In the following, further embodiments of the inventive concept for the method, which can equally be used for the corresponding maintenance monitoring system, are presented:

[0026] According to an advantageous embodiment of the method, the image analysis system can comprise a learning system with machine learning properties. The method can additionally comprise training the learning system with a set of digital images by means of machine learning to generate a learning model for detecting the at least one feature—for example, anomalies or other abnormalities—in one of the acquired digital images of samples during regular operation of the microscope system. The learning system can be, for example, a system for supervised learning, such as a deep neural network (DNN) or another suitable system for artificial intelligence or machine learning. The images of the set of digital images for supervised learning to generate the learning model can be annotated with metadata for this purpose, i.e.The respective images in the set of digital images for supervised learning are supplemented with metadata that describes, for example, anomalies or other peculiarities and features in the training images. Suggested maintenance actions based on the peculiarities, anomalies, and features in the images can also be supplemented in the metadata.

[0027] According to a supplementary embodiment of the method, the metadata can accordingly comprise at least one selected from the group consisting of: a time interval value in which the maintenance measure is required, an expected failure time of a component of the microscope system, a numerical status indication—for example, rating levels 1 to 10—of an individual component (or group) of the microscope system, and a designation of a defective component of the microscope system. In this way, the learning system is trained by the method to determine, from certain features of the captured images during regular operation of the microscope system, maintenance measures required for individual components or groups of components of the microscope system.

[0028] According to a further supplementary embodiment of the method, determining the component group of the microscope system may additionally include determining a time interval value in which the maintenance measure is required. This allows future maintenance measures to be dynamically stored in a maintenance plan. Similar maintenance measures or maintenance measures on microscope systems at different locations can thus be better coordinated by maintenance personnel.

[0029] According to an extended embodiment of the method, when the status signal is generated by the image analysis system, at least one additional sensor value - in particular not from the image sensor - of the microscope system can be recorded, which is / are used as additional input data for the learning system. It is useful to also take such sensor measured values ​​into account when training the learning system. For this purpose, it can be helpful to annotate the input data sets of the additional sensor values ​​additionally made available to the learning system accordingly, so that maintenance measures can be derived from a combination of the sensor values ​​together with the analyzed recorded images during regular operation. As an example of such additional sensor measured values, intensity values ​​of the illumination - either passive, i.e., via a sensor, or active, i.e., for example by measuring the current flow through the lamp.

[0030] According to a further advantageous embodiment of the method, the at least one feature contained in the at least one recorded digital image during regular operation includes at least one feature that is included in the group consisting of the following: stripes, jitter, uneven brightness—for example, due to uneven field illumination—reduced resolution, excessive image noise—for example, derived from one or more predefined or dynamically determined thresholds—low signal-to-background ratio, low laser power, reduced sensitivity, and defects in the digital image. In principle, these can be all detectable and executable parameters of a digital image.

[0031] According to one possible embodiment of the method, the learning model can be generated for a single microscope system or for a plurality of microscope systems. This allows the learning model to be used, for example, for all microscope systems of a series belonging to a microscope series without having to retrain it each time.

[0032] According to a supplementary embodiment of the method, during regular operation of the microscope system, at least one operating parameter value – ideally as a time series – of an operating element of the microscope system can be measured, thereby capturing user behavior. This value is additionally used to determine the status signal or influences it. For example, it can be detected that manual readjustments are repeatedly made to individual actuators of the microscope system during regular operation. These readjustments differ in number and time intervals from normal operation of the microscope system and are therefore recognized as anomalies or conspicuous features generated by sensor measurements during proper, normal operation.

[0033] Such additional recording of a user's typical operating behavior with regard to the microscope system could fall into the area of ​​unsupervised learning and training of the learning system. For this purpose, the learning system is adapted to detect abnormalities and anomalies in the recorded time series of measured parameter values ​​of individual operating elements.

[0034] According to a further developed embodiment of the method, the learning model for recognizing the at least one feature can involve at least one category selected from the group consisting of: classification of service levels of the microscope system, regression on the expected failure time of the microscope system, correlation of the at least one feature, or a combination of several features of the digital image with determination of a maintenance requirement of an assignable component or component group of the microscope system. In this way, detailed maintenance plans can be easily created and easily planned by maintenance personnel.

[0035] According to an extended embodiment of the method, the status signal can be generated from a combination of control signals. In this case, one of the control signals can relate to at least one of the following: a source of error in the microscope system, a confidence factor of the signal of the source of error, a type of error—in particular, a defect or wear—a numerical equivalent (e.g., a percentage) of a loss of quality, a required maintenance measure, a time interval for a recommended or required maintenance measure, an indication of the component for which the maintenance measure is or will be required, information—in particular, a recommendation—about alternative microscope system settings to at least partially compensate for a malfunction of a component, information about a source of error in positioning or incorrect preparation of the sample, and the like.In principle, all components and setting options - especially those components that are subject to regular maintenance - can be recorded in this way.

[0036] According to an additional, expanded embodiment of the method, the status signal can be determined by the learning system if a comparison of time sequences recorded during regular operation of a combination of sensor parameters of operating elements of the microscope system with expected values ​​of the combination of sensor parameters of operating elements of the microscope system results in a significant deviation—for example, greater than a dynamically determined threshold value. Such time sequences can also be used by the learning system—in particular through unsupervised learning during regular operation of the microscope system—to assess the sensor parameters, identify deviations, and determine or recommend maintenance measures or maintenance times.The deviation(s) to be detected from values ​​within the normal range (positive examples) can be determined by systems and / or techniques such as AutoEncoder, OneClassClassification and Novelty Detection.

[0037] According to a further possible embodiment of the method, the determination of at least one feature by the image analysis system can be determined by detecting, in a sequence of recorded images of the microscope system during regular operation, a repetitive feature which has a gradient or oscillations or sudden, irregular deviations in the chronologically ordered sequence of recorded images, or which occurs identically in each image (faulty), so that a systematic deviation from normal values ​​or anomalies becomes recognizable.

[0038] The determination of a normal value - or several - can be carried out by unsupervised learning before the microscope system is actually used or during regular operation of the microscope.

[0039] The detected gradient can have either a positive or a negative value. For example, a laser that weakens over time would result in a negative gradient in the illumination values. In this way, even gradual deterioration that would not normally be immediately noticeable to the average user of the microscope system can be elegantly detected. What was said for the laser also applies to other sensor values ​​or features in the captured digital images. This would also make it possible to detect systematic errors that would not normally be noticeable to a human user, such as individual defective pixels on the image sensor, a slightly tilted aperture, and the like.

[0040] An example of vibrations occurring could be that a control unit for the main image sensor of the microscope system no longer functions properly, which can lead to vibration-like irregularities in the recorded digital images over time.

[0041] An example of erratic behavior could be a defective lighting with a loose connection, causing the intensity of the illumination of the samples to vary - possibly even without the human eye detecting it.

[0042] According to a further extended possible embodiment of the method, the sequence of images acquired by the microscope system during regular operation can be a selected subset from a total of images acquired by the microscope system during regular operation. In this way, various filters can be used for the selection. This could include one image per day, images from the last few days, or images relating to a specific image content or application, e.g., images of specific other elements from cells such as entire cells, cell nuclei, cell walls, etc.

[0043] The acquired images could also be selected based on the metadata stored with the acquired images. The metadata for each acquisition can include, for example, the time / date, user identification of the respective operator, dyes used for the samples, sample type, type and number of color channels used, light path setting parameters, experiment class (or a complete related dataset), general experiment data (via full-text search), etc.

[0044] According to an advantageous embodiment of the method, the learning system can be a neural network—or, for example, a support vector machine. The neural network can be trained using supervised learning or unsupervised learning. The learning then allows classification, detection, and / or regression (e.g., to predict a future maintenance time). Multiclass or one-class algorithms can be used for classification. The neural network can be implemented as a deep neural network, such as a CNN (convolutional NN) or RNN (recursive NN).

[0045] In the case of unsupervised learning, algorithms such as deep autoencoders, one-class classification, multi-class classification, or novelty detection can be used. In principle, any learning system that allows the detection and determination of anomalies and trained features in the acquired digital images during regular operation can be used. Overview of the characters

[0046] It should be noted that embodiments of the invention may be described with reference to different implementation categories. In particular, some embodiments are described with reference to a method, while other embodiments may be described in the context of corresponding devices. Regardless of this, a person skilled in the art will be able to recognize and combine possible combinations of the features of the method, as well as possible combinations of features with the corresponding system, from the above and following descriptions—unless otherwise indicated.

[0047] Aspects already described above as well as additional aspects of the present invention result, inter alia, from the described embodiments and from the additional further concrete embodiments described with reference to the figures.

[0048] Preferred embodiments of the present invention are described by way of example and with reference to the following figures: Fig. 1 depicts a block diagram of an embodiment of the inventive computer-implemented method for determining a need for maintenance action for a microscope system. Fig. 2 shows a block diagram of possible extensions of the Fig. 1 presented computer-implemented procedure. Fig. 3 shows an example graph with the selection of captured images. Fig. 4 depicts a block diagram of an embodiment of the maintenance monitoring system for determining a need for maintenance action for the microscope system. Fig. 5 represents a block diagram of a computer system including at least portions of the maintenance monitoring system. Detailed character description

[0049] In the context of this description, conventions, terms and / or expressions should be understood as follows:

[0050] The term "predictive maintenance" - occasionally also preventive maintenance - in the context of this document describes how maintenance measures can be optimized by performing only those measures that are required at a specific time, depending on the condition of the device - in this case, the microscope system. This does not follow a fixed schedule; rather, it dynamically determines, based on the detected condition of the microscope system, that a maintenance measure is recommended to prevent a failure of the microscope system. The decision data for deriving a corresponding signal indicating the need for maintenance of a component group or an individual component is obtained from ongoing operation or use of the microscope system. Additional maintenance measures or special settings are not required.This allows for at least the following results: (i) fewer machine downtimes within a given operating period, (ii) short repair times, and (iii) minimal impact of downtime on the operating times of the microscope system. Furthermore, spare parts supply can be optimized, meaning fewer spare and wear parts need to be kept in stock.

[0051] The term "maintenance measure" here refers to any care, repair, adjustment, or calibration measure that ensures continuous use of the microscope system and minimizes downtime. Continuous monitoring (evaluation of the captured images) during use of the microscope system allows even subtle deterioration to be easily detected.

[0052] The term "microscope system" describes an optical or optical / electronic system for magnifying and visualizing small structures. Typically, the small structures, which are captured by one or more lenses, are recorded using an image sensor, allowing the acquired images to be further processed electronically and digitally. Samples to be examined are typically displayed in magnified form on a screen or in an eyepiece. A modern high-performance microscope system also has other components, many of which are mechanical in nature and require periodic maintenance. These may include a sample stage that can be moved in all three spatial directions, apertures, a stand for supporting the sample stage, one or more illumination systems, a variety of manual and / or automated controls and adjustment elements, etc.

[0053] The term "image sensor" describes a light-sensitive electronic semiconductor component, which in microscope systems can often be designed as an area sensor in which the captured images are directly converted into pixel information.

[0054] The term "sample" describes an object to be examined, e.g. on a sample carrier such as a multiwell plate.

[0055] The term "regular operation" describes the daily or regular use of the microscope system by one or more users—or in an automated process—to directly generate useful data in the form of digital images of samples under examination. Regular operation does not include separate, additional acquisitions of digital images intended exclusively for maintenance or other calibration purposes. Rather, the digital images acquired during regular, daily operation are used to derive appropriate maintenance measures.

[0056] The term "image analysis system" describes a system—often in conjunction with appropriate software—that is generally capable of generating metadata from the pixel information of captured images. This metadata can be based on a pixel classification to segment image regions, or regression techniques can be applied, for example, to extrapolate to a specific value. Machine learning techniques can also be used.

[0057] The term "learning system" describes a hardware-supported or software system from the field of artificial intelligence that is adapted to learn from examples. This allows the system to assign, for example, the contents of images or their sub-areas to abstract concepts—for example, "good" or "bad"—or to meaning constructs—for example, a segmentation of image content—without procedurally prescribed and manually generated program instructions. Generally speaking, a mapping from input data to output data is learned.

[0058] The term "training the learning system" means that, for example, a machine learning system adjusts parameters in a neural network, for example, by partially repeatedly evaluating the sample images, so that after the training phase, even unknown images can be assigned to one or more categories with which the learning system was trained. The sample images are typically annotated—i.e., provided with metadata—in order to generate desired results based on the input images, such as instructions for maintenance measures to be taken.

[0059] In addition, it is possible, for example, to use a regression method to predict a time for a required or recommended maintenance measure or an expected time of failure.

[0060] The term "machine learning" is a fundamental concept in artificial intelligence, where, for example, statistical methods are used to give computer systems the ability to "learn." For example, certain behavioral patterns are optimized within a specific task area. The methods used enable machine learning systems to analyze data without the need for explicit procedural programming. Typically, a CNN (Convolutional Neural Network) is an example of a machine learning system. It consists of a network of nodes that act as artificial neurons and artificial connections between the artificial neurons—so-called links—where parameters—for example, weight parameters for the connection—can be assigned to the artificial connections.During neural network training, the weight parameters of the connections adjust automatically based on input signals to produce a desired result. In supervised learning, the images provided as input values ​​are supplemented with metadata (annotations) to indicate a desired output value. In unsupervised learning, such annotations are not required.

[0061] Recursive neural networks (RNNs) should also be mentioned in this context. These are also a type of deep neural network in which weight adjustment occurs recursively, allowing a structured prediction to be generated from input data of variable size. Typically, such RNNs are used for sequential input data. Just like CNNs, they employ backpropagation functions in addition to forward weight adjustment. RNNs can also be used in image analysis.

[0062] The term "convolutional neural network" describes a class of artificial neural networks based on feed-forward techniques. They are often used for image analysis with images as input data. The main component of convolutional neural networks are convolutional layers (hence the name), which enable efficient evaluation through parameter sharing.

[0063] It should also be noted that deep neural networks consist of several layers with different functions: for example, an input layer, an output layer, and one or more intermediate layers for convolution operations, the application of nonlinear functions, dimensionality reduction, normalization functions, etc. These functions can be executed in software, or special hardware components can calculate the respective function values. Combinations of hardware and software elements are also known.

[0064] The term "deep autoencoder" refers to an autoencoder system based on the principle of a DNN. The typical goal of using autoencoders is to learn a compressed representation (encoding) for a set of data and also to extract essential features. This allows them to be used for dimensionality reduction and / or anomaly detection. The actual encoder layer lies between the input layer of artificial neurons and the output layer.

[0065] The term "one-class classification" also belongs in this context. Corresponding methods are also known as "unary classification" or "class modeling," which attempt to classify certain objects in a captured digital image as a specific class among other objects. This also requires training runs. This allows anomalies to be detected in a series of digital images.

[0066] The term "novelty detection" describes a mechanism by which an "intelligent" system (e.g., an intelligent organism) can classify an incoming sensor pattern as previously unknown. This principle can also be applied to artificial neural networks. If a sensor pattern arriving at the artificial neural network does not produce an output signal with a detection probability above a predetermined (or dynamically adjusted) threshold, or produces a plurality of output signals with approximately equal detection probabilities, the incoming sensor pattern—e.g., a digital image—can be classified as having new content (novelty).

[0067] If, however, a neural network trained using known example images (e.g., configured as an autoencoder) is fed a new example image and then passes through the autoencoder, the autoencoder should be able to reconstruct the new example image at the output. This is possible because the new example image is heavily compressed as it passes through the autoencoder, and then expanded / reconstructed by the neural network. If the new input image (largely) matches the expanded / reconstructed image, the new example image corresponds to a known pattern or it shows a high degree of similarity to the contents of the training image database. If a clear difference is evident when comparing the new example image with the expanded / reconstructed image, i.e.If the reconstruction errors are significant, then the sample image contains a previously unknown distribution (image content). This means that if a neural network cannot represent the data distribution, it constitutes an anomaly compared to the known, assumed-to-be-normal database.

[0068] A detailed description of the figures is provided below. It should be understood that all details and instructions in the figures are shown schematically. First, a block diagram of an embodiment of the inventive computer-implemented method for determining a need for predictive maintenance for a microscope system is presented. Further embodiments or embodiments of the corresponding system are described below:

[0069] Fig. 1depicts a block diagram of an embodiment of the inventive computer-implemented method for determining a need for maintenance action for a microscope system. The method comprises providing 102 a microscope system having an image sensor for capturing digital images of a sample, capturing 104 digital images by means of the image sensor during regular operation of the microscope system, and providing 106 at least one captured digital image as at least one input data set to an image analysis system.

[0070] The method further comprises: determining 108 at least one feature by the image analysis system in the at least one recorded digital image, wherein the feature correlates with a malfunction of the microscope system during regular operation, generating 110 a status signal for a state of the microscope system by the image analysis system based on the at least one digital image and the at least one feature contained therein, and determining 112 by using the status signal a component group of the microscope system for which a maintenance measure is required. This required or to be performed maintenance measure is displayed to the user. This can be done in text form or as a graphic representation by displaying an image of the microscope system and the component to be maintained with a special mark. In addition, an indication of the type of maintenance measure can be output (e.g.clean the optical system, replace the lighting, etc.).

[0071] Fig. 2 shows a block diagram of possible extensions of the Fig. 1 A function pool 214 is available for this purpose, from which additional functions can be added to the method 100 according to Fig. 1 can be integrated. This includes determining 202a time interval for a required or recommended maintenance measure on the microscope system, recording 204 additional parameter values ​​from components of the microscope system, training 206 the learning system in various forms (supervised / unsupervised learning), recording 208 parameter values ​​from control elements – possibly also time-dependent – ​​using 210 a combination of parameter values ​​to generate a status signal for a determined maintenance measure for the microscope system, and using 212 the additional parameter values ​​as input data for the learning system. For this purpose, the additional parameter values ​​– advantageously as a time series – are made available to the learning system as input data. Furthermore, the function "Select a subset of the acquired images" is available in the function pool. The selection can be made using the selection parameters mentioned above.

[0072] Fig. 3shows an example graph 300 for selecting recorded images. The x-axis represents the time (time / date combination) at which the recorded images were created or on which the corresponding experiment was conducted. Exemplary experimental measurement values ​​are plotted on the y-axis. Each measurement point 308 represents a recorded image. The images can then be selected according to different criteria (see above). Depending on the selection criteria, different groups 302, 306, and 308 of recorded images result, as the exemplary circles illustrate.

[0073] Fig. 4depicts a block diagram of an embodiment of the maintenance monitoring system 400 for determining a need for maintenance action for the microscope system. The maintenance monitoring system comprises: a microscope system 402 having an image sensor 406 for capturing digital images of a sample, wherein the image sensor 406 is used to capture digital images during regular operation of the microscope system 402.

[0074] Furthermore, the maintenance monitoring system comprises an image analysis system 404 adapted to receive at least one captured digital image as at least one input data set. The image analysis system 404 is further adapted to determine at least one feature in the at least one captured digital image, wherein the feature correlates with a malfunction of the microscope system during regular operation, generate a status signal for a state of the microscope system by the image analysis system based on the at least one digital image and the at least one feature contained therein, and determine, by using the status signal, a component group of the microscope system for which a maintenance measure is required.

[0075] Fig. 5depicts a block diagram of a computer system that may include at least portions of the maintenance monitoring system. Embodiments of the concept proposed herein can, in principle, be used with virtually any type of computer, regardless of the platform used therein for storing and / or executing program code. Fig. 5 500 illustrates, by way of example, a computer system 500 suitable for executing program code according to the method presented here. A computer system already present in the microscope system can also serve—if necessary with appropriate extensions—as a computer system for executing the concept presented here.

[0076] Computer system 500 has a plurality of general-purpose functions. The computer system may be a tablet computer, a laptop / notebook computer, another portable or mobile electronic device, a microprocessor system, a microprocessor-based system, a smartphone, or a computer system with specially configured special functions. Computer system 500 may be configured to execute computer system-executable instructions—such as program modules—that can be executed to implement functions of the concepts proposed here. For this purpose, the program modules may include routines, programs, objects, components, logic, data structures, etc., to implement specific tasks or specific abstract data types.

[0077] The components of the computer system may include one or more processors or processing units 502, a memory system 504, and a bus system 506, which connects various system components, including the memory system 504, to the processor 502. Typically, the computer system 400 includes a plurality of volatile or non-volatile storage media accessible by the computer system 400. The data and / or instructions (commands) of the storage media may be stored in the memory system 404 in volatile form—such as in a RAM (random access memory) 408—for execution by the processor 402. These data and instructions implement one or more functions of the concept presented here. Further components of the memory system 504 may include a permanent memory (ROM) 510 and a long-term memory 512, in which the program modules and data (reference numeral 516) may be stored.

[0078] The computer system has a number of dedicated devices for communication (keyboard 518, mouse / pointing device (not shown), screen 520, etc.). These dedicated devices can also be combined in a touch-sensitive display. A separately provided I / O controller 514 ensures smooth data exchange with external devices. A network adapter 522 is available for communication via a local or global network (LAN, WAN, e.g., via the Internet). The network adapter can be accessed by other components of the computer system 500 via the bus system 506. It is understood that—although not shown—other devices can also be connected to the computer system 500.

[0079] In addition, at least parts of the maintenance monitoring system 400 (see Fig. 3) may be connected to the bus system 506, such as the image sensor 406 and / or the image analysis system 404. The digital image data of the image sensor 406 may also be processed by a separate preprocessing system (not shown).

[0080] The description of the various embodiments of the present invention has been presented for clarity. The terminology used herein has been chosen to best describe the basic principles of the embodiments and to make them readily accessible to those skilled in the art.

[0081] The principle presented here can be embodied as a system, a method, combinations thereof, and / or as a computer program product. The computer program product can comprise one or more computer-readable storage media containing computer-readable program instructions for causing a processor or control system to execute various aspects of the present invention.

[0082] Electronic, magnetic, optical, electromagnetic, infrared media, or semiconductor systems can be used as the transmission medium; for example, SSDs (solid-state devices / drives), RAM (random access memory) and / or ROM (read-only memory), EEPROM (electrically erasable ROM), or any combination thereof. Other transmission media can include propagating electromagnetic waves, electromagnetic waves in waveguides or other transmission media (e.g., light pulses in optical cables), or electrical signals transmitted in wires.

[0083] The computer-readable storage medium may be an embodied device that holds or stores instructions for use by an instruction execution device. The computer-readable program instructions described herein may also be downloaded to a corresponding computer system, for example, as a (smartphone) app from a service provider via a wired connection or a cellular network.

[0084] The computer-readable program instructions for performing operations of the invention described herein may be machine-dependent or machine-independent instructions, microcode, firmware, state-defining data, or any source code or object code written, for example, in C++, Java, or similar languages, or in conventional procedural programming languages ​​such as the C programming language or similar programming languages. The computer-readable program instructions may be executed entirely by a computer system.In some embodiments, it may also be electronic circuits such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) that execute the computer-readable program instructions by utilizing status information of the computer-readable program instructions to configure or customize the electronic circuits according to aspects of the present invention.

[0085] Furthermore, the invention presented herein is illustrated with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the invention. It should be noted that virtually every block of the flowcharts and / or block diagrams may be embodied as computer-readable program instructions.

[0086] The computer-readable program instructions may be provided to a general-purpose computer, a special-purpose computer, or another programmable data processing system to produce a machine, such that the instructions, executed by the processor or computer or other programmable data processing device, generate means for implementing the functions or operations depicted in the flowchart and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium.

[0087] In this sense, each block in the illustrated flowchart or block diagrams may represent a module, segment, or portion of instructions that represents multiple executable instructions for implementing the specific logic function. REFERENCE SYMBOL

[0088] 100Procedure 102Procedure step 104Procedure step 106Procedure step 108Procedure step 110Procedure step 112Procedure step 202, ... 216Procedure steps of possible extensions 300Graph for selecting images 302, 304, 306Possible selected groups of images 308Example measured value 400Maintenance monitoring system 402Microscope system 404Image analysis system 406Image sensor 500Computer system 502Processor 504Storage system 506Bus system 508RAM 510ROM 512Long-term storage 514I / O controller 516Program modules and data 518Keyboard 520Screen 522Network adapter

Claims

1. Computer-implemented method (100) for ascertaining a need for predictive maintenance for an optical microscope system (402), the method (100) comprising - providing (102) an optical microscope system (402), which comprises an image sensor (406) for recording digital images of a sample, - recording (104) digital images using the image sensor (406) during regular operation of the optical microscope system (402), - providing (106) at least one recorded digital image as at least one input data set to an image analysis system (404), - determining (108) at least one feature by the image analysis system (404) in the at least one recorded digital image, wherein the feature correlates to a malfunction of the optical microscope system (402) during regular operation, - producing (110) a status signal for a state of the optical microscope system (402) by way of the image analysis system (404) based on the at least one digital image and the at least one feature contained therein, and - determining (112), by using the status signal, a component group of the optical microscope system (402) for which a maintenance procedure is necessary.

2. Method (100) according to Claim 1, wherein the image analysis system (402) is a learning system having properties of machine learning, and wherein the method (100) additionally comprises: - training (206) the learning system using a set of digital images by way of machine learning for generating a learning model for identifying the at least one feature in one of the recorded digital images of samples during regular operation of the microscope system (402).

3. Method (100) according to Claim 2, wherein the learning system is a system for supervised learning and, wherein the images of the set of digital images for supervised learning are annotated with metadata for generating the learning model.

4. Method (100) according to Claim 3, wherein the metadata is at least one selected from the group consisting of a time interval value, in which the maintenance procedure is necessary, an expected time of failure of a component of the microscope system (402), a numerical state indication of an individual component of the microscope system (402), and naming of a defective component of the microscope system (402).

5. Method (100) according to one of the preceding claims, wherein the determining of the component group of the microscope system (402) additionally comprises: - determining a time interval value in which the maintenance procedure is necessary.

6. Method (100) according to one of the preceding claims, wherein at least one further captured sensor value of the microscope system (402) which are used as additional input data for the learning system for producing the status signal by way of the image analysis system (404).

7. Method (100) according to one of the preceding claims, wherein the at least one contained feature in the at least one recorded digital image during regular operation is at least one from the group consisting of stripes, jitters, irregular brightness, reduced resolution, excess image noise, low signal-to-background ratio, low laser output, reduced sensitivity and imperfections in the digital image.

8. Method (100) according to one of Claims 2 to 7, wherein the learning model is generated for an individual microscope system (402) or for a plurality of microscope systems (402).

9. Method (100) according to one of the preceding claims, wherein at least one operating parameter value of an operating element of the microscope system (402) which is additionally used in the determination of the status signal is measured during regular operation of the microscope system (402).

10. Method (100) according to one of Claims 2 to 9, wherein the learning model additionally for identifying the at least one feature relates to at least one category, which is selected from the group consisting of: classification of service stages of the microscope system (402), regression to expected time of failure of the microscope system (402), correlation of the at least one feature or a combination of a plurality of features of the digital image with determination of a maintenance requirement of an assignable component or component group of the microscope system (402).

11. Method (100) according to one of the preceding claims, wherein the generation of the status signal additionally takes into account a combination of control signals, wherein in each case one of the control signals relates at least to one of the following: error source in the microscope system (402), confidence factor of the signal of the error source, type of an error, number value of a loss in quality, necessary maintenance procedure, time interval for a maintenance procedure, indication of the component for which a maintenance procedure is necessary, information relating to alternative microscope system settings to at least partially compensate a malfunction of a component, information relating to an error source of positioning or incorrect preparation of the sample.

12. Method (100) according to one of Claims 2 to 11, wherein the status signal is determined by the learning system additionally in that a comparison with time series, captured during regular operation, of a combination of sensor parameters of operating elements of the microscope system (402) to comparison values of the combination of sensor parameters of operating elements of the microscope system (402) provides a significant deviation.

13. Method (100) according to one of the preceding claims, wherein the determining of at least one feature is performed by the image analysis system (404) by way of - detecting, in a sequence of recorded images of the microscope system (402) during regular operation, a repeating feature which has in the chronologically arranged sequence of recorded images a gradient, fluctuations or an abrupt behaviour or which appears identically in each image.

14. Method (100) according to Claim 13, wherein the sequence of recorded images of the microscope system (402) during regular operation is a selected subset from a totality of recorded images of the microscope system (402) during regular operation.

15. Method according to one of Claims 2 to 14, wherein the learning system is a neural network.

16. Method according to one of Claims 13 of 15, wherein the neural network is based on at least one of the following technologies: deep autoencoder, one-class classification or novelty detection.

17. Maintenance monitoring system (400) for predictive maintenance for an optical microscope system (402), the maintenance monitoring system (400) comprising - an optical microscope system (402) comprising an image sensor (406) for recording digital images of a sample, wherein the image sensor is adapted for recording digital images during regular operation of the optical microscope system (402), - an image analysis system (404), which is adapted for receiving at least one recorded digital image as at least one input data set to the image analysis system, wherein the image analysis system (404) is adapted for - determining at least one feature by the image analysis system (404) in the at least one recorded digital image, wherein the feature correlates to a malfunction of the optical microscope system (402) during regular operation, - producing a status signal for a state of the optical microscope system (402) by way of the image analysis system (404) based on the at least one digital image and the at least one feature contained therein, and - determining, by using the status signal, a component group of the optical microscope system for which a maintenance procedure is necessary.

18. Computer program product for ascertaining a predictive maintenance for an optical microscope system (402) having an image sensor for recording digital images of a sample, wherein the computer program product includes a computer-readable storage medium, having program instructions which are stored thereon, wherein the program instructions are executable by one or more computers (400) or control units and cause the one or the plurality of computers (400) or control units to perform the following: - recording (104) digital images using the image sensor during regular operation of the optical microscope system, - transmitting (106) at least one recorded digital image as at least one input data set to an image analysis system, - determining (108) at least one feature by the image analysis system in the at least one recorded digital image, wherein the feature correlates to a malfunction of the optical microscope system during regular operation, - producing (110) a status signal for a state of the optical microscope system (402) by way of the image analysis system (404) based on the at least one digital image and the at least one feature contained therein, and - determining (112), by using the status signal, a component group of the optical microscope system (402) for which a maintenance procedure is necessary.