Method and apparatus for investigating microscope samples

Abstract

The disclosure relates to a computer-implemented method for creating a database for an analysis of at least one microscope sample with the aid of a microscopy apparatus, wherein the method comprises at least the following steps: obtaining input data relating to a detection of at least one action of an operator of the microscopy apparatus in relation to the analysis of the microscope sample with the aid of the microscopy apparatus, and relating to the microscopy apparatus and / or the microscope sample, generating database data on the basis of the input data, with the database data containing at least one relationship between the detected action of the operator and the apparatus, the analysis and / or the microscope sample, and storing the database data in the database.

Description

[0001] Method and apparatus for examining microscope samples

[0002] Technical field

[0003] The invention relates to a computer-implemented method for creating a database for an analysis of at least one microscope sample, a computer-implemented method for determining predictions concerning an analysis of a microscope sample, corresponding data processing devices, a method for operating a microscopy device for analyzing at least one microscope sample, and a microscopy system for analyzing at least one microscope sample.

[0004] background

[0005] When analyzing a microscope sample, the operator typically performs an action, such as changing a hardware setting of the microscopy device or optimizing a display during the analysis. The invention aims to make analysis using a microscopy device more efficient, versatile, and / or cost-effective.

[0006] Depiction

[0007] This objective is achieved, firstly, by a computer-implemented method for creating a database for the analysis of at least one microscope sample using a microscopy apparatus. The method comprises at least the following steps: (1) obtaining input data (a) relating to the detection of at least one action by an operator of the microscopy apparatus in relation to the analysis of the microscope sample using the microscopy apparatus, and (b) relating to the microscopy apparatus and / or the microscope sample; (2) generating database data based on the input data, wherein the database data contains at least a relationship between (a) the detected action of the operator and (b) the apparatus, the analysis, and / or the microscope sample; and (3) storing the database data in the database.

[0008] At least one step of the process is carried out using a computer.

[0009] This method can be used for the systematic construction of a database. In particular, it can be used to create training data, e.g., for an AI application or a machine learning model.

[0010] The correlation can relate to available information about the sample and the system or device settings. For example, a specific image area can be correlated with sample properties and the system settings selected by the operator. The analysis can be based on image data and / or other data, such as non-image measurement data.

[0011] The detection (recognition) of the action can involve observation, meaning the detection can extend over a specific time interval. Detection can also involve tracking.

[0012] This makes it possible to create a database that establishes relationships between the operator's actions and the device, analysis, and / or sample. Using this database, an action can be linked to the device, analysis, and / or sample in advance, or the relationship can at least partially anticipate an operator's action by predicting future operation based on the device, analysis, and / or sample. In this way, the database can form the basis for efficient, user-friendly, and / or safe operation of a microscopy device.

[0013] Optionally, the relationship includes a correlation of analysis values ​​relating to the microscope sample and / or spatial regions of image data of the microscope sample, at least one system setting of the microscope device, a parameter relating to environmental characteristics of the device, and / or an analysis result. For example, a relationship can relate to the sample characteristics and a system setting and / or an environmental parameter, such as atmospheric pressure. A correlation between an image region and an evaluation region, to which the operator refers sequentially and / or repeatedly, is conceivable. Such a correlation can arise if an image region and an evaluation region are considered subsequently and / or repeatedly. A correlation between considered data, settings, and spatial resolution is conceivable. The relationship can involve an annotation of a result with one or more parameters.Through such a correlation, e.g. with considered data, settings and spatial resolution, important information for sample characterization and associated system settings can be automatically collected and databases created that can later be used, e.g. for better use of systems, system presets, experiment planning and even experiment predictions.

[0014] Optionally, action detection is based at least partially on image recognition. Alternatively or additionally, the detected action relates at least partially to a body part of the operator, optionally an eye, and / or an acoustic signal from the operator. For example, detection can be performed using one or more cameras or eye tracking. Alternatively or additionally, detection can be based on a microphone and / or a gyroscope sensor. This can enable precise and / or reliable detection, which in turn reduces errors in recognizing correlations. By using eye-tracking microscopy control, workflows for operators can be accelerated and also made possible for operators with physical limitations, such as those unable to use their hands.

[0015] Optionally, the detected operator action, particularly in the case of image recognition or eye tracking, is based on the operator's pupil movement. This can relate to the position of the gaze and / or the time the eye spends on a specific area of ​​the device and / or the opening / closing of at least one eye. For example, a specific area of ​​the device could be a display or representation of the sample and / or an analysis thereof and / or a control element of the device. This can enable precise and / or reliable detection, which in turn reduces errors in recognizing correlations. Pupil movement can be observed or tracked, meaning it can be evaluated where the operator is looking. Gaze angles can be calculated and assigned to a point / area of ​​the device / display / control element. For example, points of interest on a sample can be identified, perhaps by the eye spending a longer time on them.

[0016] Optionally, the analysis of the sample extends, at least partially, to image data and / or analytical values ​​displayed using the device. This can lead to the creation or identification of useful correlations.

[0017] Optionally, the detected action includes the operator operating the device, adjusting a device control, and / or changing the display of the analysis. For example, hardware control is possible; for instance, the operator can move digital sliders or switches, which is used to establish a correlation. Controlling the display of the sample and / or the analysis is also conceivable. Regulating the exposure depending on antibody staining is a concrete example. By detecting device operation, the database can include operator actions related to the device and / or analysis, which can facilitate future operations.

[0018] Optionally, input data can be excluded from database generation if it relates to an operator action that is excluded from database creation and / or is unsuitable. Such data may not be useful for database construction, thus allowing for the creation of a "better" database. For example, input data can be excluded if the operator reverses an action because it was not advantageous, unintended, or incorrectly identified. The objective is also achieved through a computer-implemented method for generating predictions regarding the analysis of a microscope sample using a microscopy device and a database.The method comprises (1) receiving at least one input relating to the microscope sample, analysis, and / or microscopy apparatus, and (2) generating output data based on the input and the database, optionally generated according to the inventive method for creating the database. The output data relates to the analysis of the microscope sample using the microscopy apparatus and is based on at least one relationship stored in the database between (a) an action by an operator of the microscopy apparatus and (b) the microscope sample, microscopy apparatus, and / or analysis. The method further comprises (3) determining at least one predictive value relating to the analysis, microscope sample, and / or microscopy apparatus based on the output data.

[0019] This makes it possible to use relationships stored in the database for prediction, thereby simplifying, accelerating (possibly anticipating or even rendering obsolete) and / or making analysis more user-friendly.

[0020] At least one step of the process is carried out using a computer.

[0021] The input can contain text, image, and / or audio information. In particular, the input can be a microscope sample.

[0022] Optionally, the predictive value refers to the analysis, forecasting, planning, simulation, and / or control of the analysis of the microscope sample using the microscopy apparatus. The prediction or predictive value can relate to the creation of forecasts or hypotheses, or the simulation of results. The prediction or predictive value can be used for experiment planning, for suggesting materials and samples to be used, and / or suitable system settings. The prediction can involve planning future investigations, providing advice on settings, and / or suggesting highlighting in data, e.g., image areas. Using the prediction, it is possible, for example, to extrapolate results to another organism (from mouse to human), whereby certain pathways can potentially be successfully predicted species-specifically with the help of the database.

[0023] The prediction can refer to (virtual) forecasts of investigations or analyses. The prediction can represent a simulation, i.e., a virtual analysis / investigation performed entirely on a computer ("in silico"). The prediction can refer to a value that aids in the search for suitable investigation parameters: proteins as biomarkers, transcriptional information, and deviations in phenotypes and genotypes of specific cell types are examples of this.

[0024] The database can be a self-created database, e.g. according to the inventive method, and / or an external database.

[0025] The predicted value can be used and / or output as part of the forecasting process.

[0026] Optionally, the predicted value can include operating instructions for the operator to use the microscopy device to analyze the microscope sample. This can make the operation of the device more user-friendly.

[0027] Optionally, the output data is generated using an AI-based algorithm or a machine learning model. The AI-based algorithm can learn and thus potentially recognize new relationships in the stored database data.

[0028] The invention further relates to a data processing device with one or more processors and one or more storage devices, wherein the processor(s) is / are configured to carry out the inventive method for creating the database and / or for determining the prediction.

[0029] The invention further relates to a computer program product comprising instructions which, when the program is executed by a computer, cause it to execute the inventive method for creating the database and / or for determining a prediction.

[0030] The invention further relates to a computer-readable data carrier on which the computer program product according to the invention is stored.

[0031] A data processing device can include a database and be a computer system.

[0032] The goal is further achieved by a method for operating a microscopy device for analyzing at least one microscope sample.The method comprises at least the following steps: (1) providing a microscope sample for analysis using the device, (2) detecting an operator action during the analysis of the microscope sample and identifying the action in relation to the analysis, sample and / or the microscopy device, (3) at least partially operating the microscopy device during the analysis of the microscope sample based on the operator action, and (4A) at least partially storing data relating to the operator action, the analysis, an analysis result, the microscope sample and / or the microscopy device, and / or (4B) suggesting an operation and / or evaluation to the operator relating to the analysis, the analysis result, the microscope sample and / or the microscopy device based on the detected action.

[0033] The procedure may involve a microscopy procedure with the execution of microscopy workflows.

[0034] The method can relate to indirect control of the device by observing the action, such that the control is not directly triggered by the action itself, but by converting the action into another / virtual action that initiates the control. The method according to the invention can be carried out without the use of input aids such as a mouse, joystick, or keyboard. It can be a contactless control, in which the operator does not exert any physical contact with the device, i.e., control without physical interaction with operating elements, i.e., with non-contact control means.

[0035] The device may have a motorized or automated system.

[0036] Detection or observation can refer to speech ("chatting"), gestures, eyes or other means, possibly in combination.

[0037] This method can be advantageously suited for operators with physical limitations, it can support rapid operation and / or evaluation, and / or it can enable contactless control. For example, microscopy systems or devices in high-security laboratories can be operated safely without having to touch them.

[0038] Optional data storage can be advantageous if it results in the creation of a database system, e.g., according to the invention. A suggested user interface can accelerate and / or improve operation and / or evaluation.

[0039] Optionally, a predictive value regarding the analysis, microscope sample, and / or microscopy apparatus is proposed, further optionally according to the method of the invention. This can accelerate and / or improve operation and / or evaluation.

[0040] Optionally, the microscopy device controls and / or an operating suggestion is performed automatically after a confirming action by the operator. This can speed up and / or improve operation and / or evaluation.

[0041] Optionally, the microscopy device can be operated by virtually activating a control element. This can simplify operation and, for example, make devices in inaccessible areas usable. The control element can be used for image acquisition and / or analysis.

[0042] Optionally, the duration, repetition, frequency, and / or amplitude of operator actions can be used to identify the action. This can support precise, fast, and error-free identification.

[0043] Optionally, action detection is based at least partially on image recognition. Alternatively or additionally, the detected action relates at least partially to a body part of the operator, optionally an eye, and / or an acoustic signal from the operator. For example, detection can be performed using one or more cameras or eye tracking. Alternatively or additionally, detection can be based on a microphone and / or a gyroscope sensor. This can enable precise and / or reliable detection, which in turn reduces errors in recognizing a correlation.

[0044] Optionally, the detected operator action, particularly in the case of image recognition and eye tracking, is based on the operator's pupil movement. This can relate to the position of the gaze and / or the eye's dwelling on a specific area of ​​the device and / or the opening / closing of at least one eye.

[0045] For example, a section of the device can be a display or representation of the sample and / or an analysis thereof and / or a control element of the device. This can enable precise and / or reliable detection, which in turn reduces errors in recognizing correlations. Pupil movement can be observed or tracked, meaning it can be evaluated where the operator is looking. Viewing angles can be calculated and assigned to a point / section of the device / display / control element. For example, points of interest on a sample can be identified, e.g., by the eye dwelling on them for a longer period of time.

[0046] Optionally, the data are stored in a database for the purpose of generating database data, possibly according to the method according to the invention. Automatic storage can mean that this occurs without active intervention from the operator. For example, the database can be automatically populated during the operator's analysis of the sample, so that no separate analysis is necessary solely for training purposes. Optionally, data will not be stored if it relates to an operator action that is excluded from database generation and / or is unsuitable. This can lead to a "better" database.

[0047] Optionally, data is stored that improves the identification of an operator's action based on the detected action and / or provides feedback on operator satisfaction. This can improve operation in terms of user-friendliness and / or precision.

[0048] The invention further relates to a microscopy system for analyzing at least one microscope sample. The system comprises at least one receptacle for receiving the microscope sample, a microscopy device for at least partial analysis of the microscope sample by the operator, a detector for detecting at least one action of the operator relating to the microscopy device and / or the analysis, and one or more units that are / are configured to carry out the inventive method for operation.

[0049] A microscope sample can consist of an object to be examined under the microscope and a sample holder to hold the object.

[0050] The microscopy apparatus may include a computer-based microscope; a light microscope; an electron microscope; a device for processing the object; a device for freezing the microscope sample; and / or a cryocontainer for at least one microscope sample.

[0051] Optionally, the microscopy system further comprises a data processing device according to the invention. This enables a system configured to operate a microscopy device, thereby generating a database, and thus determining a prediction. In other words, the system can be configured for database generation and / or prediction determination.

[0052] Optional embodiments of the device and method according to the invention are explained by way of example. The individual embodiments can be combined and interchanged independently of one another. In particular, it is also possible for the device to be configured to implement a feature specified below as a process step. Conversely, a functional suitability of the device can also constitute a process step. Images of the sample can be acquired using different modalities, for example, visible and / or invisible light, fluorescent light, and / or electron beam.The images can alternatively or additionally be acquired using different contrast and / or recording methods, for example brightfield microscopy, darkfield microscopy, fluorescence microscopy, confocal microscopy including spinning disc, light sheet microscopy, multiphoton microscopy, high-resolution microscopy such as STED or localization microscopy, electron microscopy and / or atomic force microscopy.

[0053] A microscopy device can be a microscope. A microscopy system can include a microscopy device, i.e., a microscope.

[0054] The device can, for example, be configured to produce at least one image using a light microscope and / or at least one image using an electron microscope. The operating procedure can include the process step of producing at least one image using a light microscope and / or producing at least one image using an electron microscope and / or an atomic force microscope.

[0055] The device may include at least one device from the following list: at least one light microscope; at least one electron microscope; at least one atomic force microscope; at least one device for processing the object; at least one device for freezing the microscope sample; at least one cryocontainer for at least one microscope sample; at least one microtome; at least one micromanipulator; at least one laser microdissection unit; at least one optical tweezers; at least one high-pressure freezer; at least one grid plunger; at least one dry shipper, for example a dry shipping container and / or a nitrogen transport container.

[0056] The light microscope can be configured as a slide scanner, a high-resolution microscope such as a STED or a localization microscope, a stereo microscope, a phase-difference microscope, a transmitted light microscope, an reflected light microscope, a bright-field microscope, a dark-field microscope, a light-sheet microscope, a laser microdissection microscope, and / or a confocal microscope, including combinations of such microscopes. The device can also include several such light microscopes, even of different types, with which the microscope sample is examined sequentially.

[0057] The light microscope is specifically designed for examining frozen microscope samples. The electron microscope can be a scanning and / or transmissive electron microscope. The electron microscope can also be specifically designed for examining frozen microscope samples.

[0058] The device may, in particular, include a processor, for example, an image data processing processor, which may be configured, for example, as a CPU, GPU, vector processing unit (VPU), ASIC, FPGA, and / or any combination of such components, including memory. The process steps described above may be implemented as hardware and / or software components.

[0059] The invention is described in more detail below using an exemplary embodiment. As explained above, features of the embodiment can be omitted if the technical effect associated with these features is not important for a particular application. Conversely, further features can be added to the embodiment if their technical effect is important for a particular application.

[0060] In the following, the same reference symbols are used for features that correspond to each other in terms of function and / or spatial design.

[0061] Brief description of the drawings

[0062] Fig. 1 shows a schematic representation of a system according to the invention for carrying out a method according to the invention.

[0063] Fig. 2 shows a schematic representation of a system according to the invention during the execution of a method according to the invention.

[0064] Fig. 3 shows a representation of the steps involved in carrying out a method according to the invention.

[0065] Detailed description

[0066] Figure 1 shows a schematic diagram of a microscopy system 100 for analyzing a microscope sample 106. The system 100 has a receptacle 108 for receiving the microscope sample 106. Furthermore, the system 100 has a microscopy device (microscope) 101 for at least partial analysis of the microscope sample 106 by the operator (i.e., the user), and a detector 107 for detecting at least one action of the operator relating to the microscopy device 101 and / or the analysis. The detector 107 serves for image acquisition or as an eye tracker, i.e., for observing the pupil of at least one eye of the operator.

[0067] System 100 further comprises a display 102 (screen) for the visual representation of sample 106, a display 103 for system settings (right) and analysis results (left), a device control unit 104, and a data processing unit 105 (computer) with a database. Specifically, display 103 shows, on the right, a section of a monitor image with system setting data (e.g., light intensity, camera exposure times, contrast method, magnification, table position, etc.) and, on the left, a display of the analysis results of sample 106. Screen 102 shows the visualization of sample 106. The data processing unit 105 includes control boxes and the PC, each with buttons and indicators (e.g., rotary knobs, LEDs, switches, etc.).

[0068] The control of the microscopy device 101 and / or an operating suggestion can be carried out automatically after a confirming action by the operator.

[0069] Fig. 1 thus shows the general setup for control via eye tracking and simultaneously for the use of recorded eye tracker data for the automatic creation of experiment sequences and settings in combination with recorded result data and enables automatic annotation or linking of the result data by evaluating the eye tracking.

[0070] Figure 2 shows eye-tracker positions, or points 201 to 204, of interest where the operator's eye rests during the use (operation or evaluation) of system 100. The arrows reflect the sequence of actions. The duration, frequency, and / or amplitude of the eye or pupil can be used to identify the action. In addition to the spatial information, i.e., the relevant areas in the image field, as displayed by indicator 102, the operator's viewing angle changes can also be determined.

[0071] The eye tracker position 203 on the screen 102, the eye tracker positions 201 and 202 on the display 103, and the eye tracker position 204 on the device control 104 can be used to build a database on a computer 105. In particular, when looking at the screen 102, it is possible to distinguish between areas showing the sample 106 and to determine and record the spatial information, i.e., the X, Y, Z position within the sample 106. It is assumed that areas of interest in the sample are viewed for longer than areas that are irrelevant. In addition to the position and duration of the eye or pupil, the (repeated) opening and closing of the eye and its duration can also be evaluated and used as a control command.

[0072] Since the visual representation of sample 106 on screen 102 (or, if applicable, via eyepieces) allows the determination of sample 106's coordinates, the relevant areas viewed for extended periods can also be determined in X, Y, and Z coordinates. These areas can be correlated with all available information about sample 106 and all setting data of the imaging microscope system 100. This correlation reflects a relationship that is stored in the database. This relationship is based on input data concerning the detection of at least one action by an operator of the microscopy device 101, i.e., the operator's pupil movement, in relation to the analysis of the microscope sample 106 using the microscopy device 101, and input data concerning the microscopy device 101 and / or the microscope sample 106 itself.This generates database data, the database data containing at least a relationship between the detected action of the operator and the device 101, the analysis and / or the microscope sample 106.

[0073] Once sufficient information has been gathered—that is, at least one experiment or analysis has been completed and data stored in the database—this data can be used to plan future experiments, advise the user on settings, and, if necessary, directly suggest important data and image areas by highlighting them. Databases can be used for analysis, forecasting, and suggestion software; system settings; experiment planning; and predicting experiment outcomes.

[0074] To meaningfully expand the database, eye tracking can be restricted or suspended in certain situations to prevent potentially incorrect or erroneously irrelevant highlighted information from further triggering the database and thus filling it up. This might be the case, for example, with "suggestion displays."

[0075] For example, if the user looks at analysis values ​​or graphs and then directly at a specific area of ​​the image, and repeats this process several times, the corresponding analysis data can be contextualized with the "spatial information," i.e., a specific sub-area of ​​the image. This information can also be stored in the database. These areas can be correlated with all available information about sample 106 and all settings of the imaging (microscope) system 100. This correlated data is stored in the database and continuously collected during system use, thus automatically expanding database 105. In this way, a connection can be established between the eye tracker and the viewed data, regions, and settings, and the database can be generated.

[0076] With an increasing amount of relevant data, and potentially also a "negative database"—that is, a collection of data from less interesting areas or analysis results to be excluded—predictions of results or analyses can be generated purely virtually, and / or the planning and search for relevant candidates such as proteins as biomarkers, transcriptional information, and variations in phenotypes and genotypes of specific cell types can be calculated. These predictions can then be used without any experimentation, purely in silico, and with the aid of and correlation with external, public databases, to forecast targeted studies and experiments.

[0077] Data can be stored to improve the identification of an operator's action based on the detected action and / or to obtain feedback on operator satisfaction.

[0078] Figure 2 illustrates the application in eye tracking, where the dashed circles indicate the gaze positions 201 to 204, which are determined by the detector 107 using an eye tracker and where the operator's gaze remains fixed (e.g., for at least 200 ms, which can be individually adjusted for the operator). The dashed arrows show the movement of the eye from one "fixed point" to the next. In this example, two analysis points 201 and 202, a specific position 203 on the visualized sample 106, and a rotary knob setting 204 of a control box 104 are considered. It can be assumed that the analysis / result values ​​belong to the considered sample position, and thus the sample 106 is annotated at this point with the corresponding result values. Optionally, this can also be linked to an audio input. The position of the rotary knob (e.g.,Fluorescence intensity can also be stored for this annotated result along with the digitally available setting parameters for sample 106. All data are annotated and linked in the database. Accordingly, digital sliders or switches (not shown) can be set with the eye tracker without physical interaction with the system 100.

[0079] Figure 3 schematically illustrates the prediction process. In step 301, data is generated and stored based on system 100. The internally generated database is imported in step 302. Alternatively or additionally, an external database can be imported in step 303. In step 304, input is made regarding the microscope sample 106, analysis, and / or microscopy device 101. This can be text input or another sample 106 to be examined in system 100. Output data is then generated using the database from step 302 and / or the external database from step 303.In step 305, the output data relating to the analysis of microscope sample 106 using microscopy device 101 are determined and are based on at least one relationship stored in the database between an action by an operator of the microscopy device 101 and the microscope sample 106, microscopy device 101, and / or analysis. A predictive value relating to the analysis, microscope sample 106, and / or microscopy device 101 is then calculated.

[0080] The predicted value in step 305 can be output or saved and can relate to the analysis, forecasting, planning, simulation, and / or control of the analysis of microscope sample 106 using the microscopy device 101. Alternatively or additionally, the predicted value can relate to an operating suggestion for the operation of the microscopy device 101 for the analysis of sample 106.

[0081] In other words, the results from Figure 2 (step 301) are stored in a local database (see Figure 3). These stored results can then be used to plan further experiments (step 305). Furthermore, if sufficient data is available (e.g., at least 10 similar experiments, or x to n experiments depending on the complexity of the experiment), the collected results in the internal database (step 302) can be linked, if necessary, with other internal or external databases (step 303) in order to simulate results, make predictions, or formulate hypotheses in step 305.

[0082] System 100 is used to examine a microscope sample 106. The microscope sample 106 contains an object (not shown). The object is preferably a biological object, for example, a part of an organism prepared for microscopy, such as a tissue section, or individual cells. However, the object can also be inorganic and / or contain or consist of inorganic components.

[0083] A sample holder (not shown) for the microscope sample 106 is preferably made of an optically transparent material, which is further preferably also transparent to electron beams. In particular, the sample holder can be made of glass or metal.

[0084] To generate a digital image, the System 100 can include a light microscope and / or an electron microscope as a microscopy device 101. The light microscope can be a slide scanner, a high-resolution microscope such as a STED microscope or a localization microscope, a stereo microscope, a phase-difference microscope, a bright-field microscope, a dark-field microscope, a fluorescence microscope, an incident-light microscope, a transmitted-light microscope, a light-sheet microscope, and / or a confocal microscope. The System 100 can include several light microscopes of the same or different configurations. The electron microscope can be a scanning and / or a transmissive electron microscope. The System 100 can include one or more electron microscopes of the same or different designs. Instead of or in addition to the electron microscope, an atomic force microscope can be used or included in the System 100.

[0085] Both the at least one light microscope and the at least one electron microscope are preferably designed for examining frozen microscope samples, so-called cryo-samples. For this purpose, the at least one light or electron microscope 101 can have a region for receiving the sample holder, which is cooled and / or thermally insulated.

[0086] System 100 may further comprise at least one device (not shown) for processing the object. Such a device is, for example, a robot-controlled tweezer, a picker, a pipette, a microtome, a micromanipulator, optical tweezers, an ion beam ablator and / or a dissection device such as a laser microdissection device, or a combination thereof.

[0087] In the event that cryopreserved samples are to be examined with the system 100, at least one device for freezing the microscope sample 106 may be provided. Such a device may be, for example, a refrigerator, a high-pressure freezer, and / or a grid plunger. The device may be integrated into a microscopy apparatus 101.

[0088] Some embodiments relate to a microscope 101 or a system 100, as described in connection with Figures 1 and 2. Alternatively, a microscope can be part of, or connected to, a system 100, as described in connection with one or more of Figures 1 and 2. Figures 1 and 2 show a schematic representation of a system 100 configured to perform a method described herein. The system 100 comprises a microscope 101 and a computer system 105. The microscope 101 is configured to acquire images and is connected to the computer system 105. The computer system 105 is configured to perform at least part of a method described herein. The computer system 105 can be configured to execute a machine learning algorithm. The computer system 105 and the microscope 100 can be separate units or can be integrated together in a common housing.The computer system 105 could be part of a central processing system of the microscope 101 and / or the computer system 105 could be part of a subcomponent of the microscope 101, such as a sensor, an actuator, a camera or a lighting unit, etc. of the microscope 101.

[0089] The computer system 105 can be a local computing device (e.g., a personal computer, laptop, tablet computer, or mobile phone) with one or more processors and one or more storage devices, or it can be a distributed computing system (e.g., a cloud computing system with one or more processors or one or more storage devices distributed at different locations, for example, at a local client and / or one or more remote server farms and / or data centers). The computer system 105 can comprise any circuit or combination of circuits. In one embodiment, the computer system 105 can comprise one or more processors, which can be of any type.In our usage, "processor" can mean any type of computing circuit, such as, but not limited to, a microprocessor, a microcontroller, a complex instruction set (CISC) microprocessor, a reduced instruction set (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field-programmable gate array (FPGA), e.g., of a microscope or a microscope component (e.g., a camera), or any other type of processor or processing circuit. Other types of circuits that may be included in Computer System 105 may be a custom-made circuit, an application-specific integrated circuit (ASIC), or the like, such as one or more circuits (e.g., a communications circuit) for use in wireless devices, such as...The computer system 105 may include mobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The computer system 105 may include one or more storage devices, which may comprise one or more memory elements suitable for the application, such as main memory in the form of random access memory (RAM), one or more hard disks, and / or one or more drives that handle removable media, such as CDs, flash memory cards, DVDs, and the like. The computer system 105 may also include a display device, one or more speakers, and a keyboard and / or control device, which may include a mouse, trackball, touchscreen, voice recognition device, or any other device that allows a system user to input information into and receive information from the computer system 105.

[0090] Some or all of the process steps can be performed by (or using) a hardware device, such as a processor, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the key process steps can be performed by such a device.

[0091] Depending on specific implementation requirements, embodiments of the invention can be implemented in hardware or software. Implementation can be carried out using a non-volatile storage medium such as a digital storage medium, for example, a floppy disk, DVD, Blu-ray disc, CD, ROM, PROM, EPROM, EEPROM, or FLASH memory, on which electronically readable control signals are stored. These signals interact (or can interact) with a programmable computer system to execute the respective method. Therefore, the digital storage medium can be computer-readable.

[0092] Some embodiments according to the invention include a data carrier with electronically readable control signals that can interact with a programmable computer system so that one of the methods described herein is carried out.

[0093] In general, embodiments of the present invention can be implemented as a computer program product with program code, wherein the program code is effective for executing one of the methods when the computer program product runs on a computer. The program code can, for example, be stored on a machine-readable medium.

[0094] Further embodiments include the computer program for carrying out one of the methods described herein, which is stored on a machine-readable medium.

[0095] In other words, an embodiment of the present invention is therefore a computer program with program code for carrying out one of the methods described herein when the computer program is running on a computer.

[0096] Another embodiment of the present invention is therefore a storage medium (or a data carrier or a computer-readable medium) comprising a computer program stored thereon for executing one of the methods described herein when executed by a processor. The data carrier, the digital storage medium, or the recorded medium is generally tangible and / or not seamless. Another embodiment of the present invention is a device as described herein comprising a processor and the storage medium. Another embodiment of the invention is therefore a data stream or signal sequence representing the computer program for carrying out one of the methods described herein. The data stream or signal sequence can, for example, be configured to be transmitted via a data communication link, such as the Internet.

[0097] Another embodiment includes a processing means, for example a computer or a programmable logic device, which is configured or adapted to perform one of the methods described herein.

[0098] Another embodiment comprises a computer on which the computer program for performing one of the methods described herein is installed.

[0099] Another embodiment according to the invention comprises a device or system configured to transmit (for example, electronically or optically) a computer program for executing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, or the like. The device or system may, for example, include a file server for transmitting the computer program to the receiver. In some embodiments, a programmable logic device (e.g., a field-programmable gate array, FPGA) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field-programmable gate array may be combined with a

[0100] Microprocessors cooperate to perform one of the procedures described herein. In general, the procedures are preferably performed by each hardware device.

[0101] Examples of implementation can be based on the use of a machine learning model or machine learning algorithm. Machine learning can refer to algorithms and statistical models that computer systems can use to perform a specific task without the use of explicit instructions, instead of relying on models and inference. For example, instead of a rule-based transformation of data, machine learning can use a transformation of data derived from an analysis of historical and / or training data. For instance, the content of images can be analyzed using a machine learning model or a machine learning algorithm. To enable the machine learning model to analyze the content of an image, it can be trained using training images as input and training content information as output.By training the machine learning model with a large number of training images and / or training sequences (e.g., words or sentences) and associated training content information (e.g., labels or annotations), the machine learning model "learns" to recognize the content of the images. This allows the machine learning model to utilize the content of images not included in the training data. Furthermore, by training a machine learning model using training sensor data and a desired output, the machine learning model "learns" a conversion between the sensor data and the output. This can be used to provide an output based on non-training sensor data provided to the machine learning model. The provided data (e.g., sensor data, metadata, and / or image data) can be preprocessed to obtain a feature vector, which is then used as input for the machine learning model.

[0102] Machine learning models can be trained using training input data. The examples above use a training method called supervised learning. In supervised learning, the machine learning model is trained using a plurality of training samples, where each sample can include a plurality of input data values ​​and a plurality of desired output values; that is, each training sample is associated with a desired output value. By providing both training samples and desired output values, the machine learning model "learns" which output value to provide based on an input sample that is similar to the samples provided during training. In addition to supervised learning, semi-supervised learning can also be used. In semi-supervised learning, some of the training samples lack a desired output value.Supervised learning can be based on a supervised learning algorithm (e.g., a classification algorithm, a regression algorithm, or a similarity learning algorithm).

[0103] Classification algorithms can be used when the outputs are restricted to a limited set of values ​​(categorical variables), i.e., the input is classified as one from the limited set of values. Regression algorithms can be used when the outputs exhibit any numerical value (within a range). Similarity learning algorithms can be similar to both classification and regression algorithms, but they are based on learning from examples using a similarity function that measures how similar or related two objects are. In addition to supervised or semi-supervised learning, unsupervised learning can be used to train the machine learning model. In unsupervised learning, input data may be provided (only), and an unsupervised learning algorithm can be used to find a structure in the input data (e.g.,(by grouping or clustering the input data, finding commonalities in the data). Clustering is the assignment of input data, which comprises a plurality of input values, into subsets (clusters) such that input values ​​within the same cluster are similar according to one or more (predefined) similarity criteria, while they are dissimilar to input values ​​that are included in other clusters.

[0104] Reinforcement learning is a third group of machine learning algorithms. In other words, reinforcement learning can be used to train the machine learning model. In reinforcement learning, one or more software actors (so-called "software agents") are trained to perform actions in an environment. A reward is calculated based on the actions performed. Reinforcement learning relies on training the one or more software agents to select actions in such a way that the cumulative reward increases, resulting in software agents that become better at the task they are given (as evidenced by increasing rewards).

[0105] Furthermore, some techniques can be applied to certain machine learning algorithms. For example, feature learning can be used. In other words, the machine learning model can be trained, at least partially, using feature learning, and / or the machine learning algorithm can include a feature learning component. Feature learning algorithms, also called representation learning algorithms, can preserve the information in their input but transform it in a way that makes it useful, often as a preprocessing stage before performing classification or prediction. Feature learning can be based, for example, on principal component analysis or cluster analysis.

[0106] In some examples, anomaly detection (i.e., outlier detection) can be used to identify input values ​​that raise suspicion because they differ significantly from the majority of input and training data. In other words, the machine learning model can be trained, at least partially, using anomaly detection, and / or the machine learning algorithm can include an anomaly detection component.

[0107] In some examples, the machine learning algorithm can use a decision tree as a predictive model. In other words, the machine learning model can be based on a decision tree. In a decision tree, observations about an object (e.g., a set of input values) can be represented by the branches of the decision tree, and an output value corresponding to the object can be represented by the leaves of the decision tree. Decision trees can support both discrete and continuous values ​​as output values. When discrete values ​​are used, the decision tree can be called a classification tree; when continuous values ​​are used, the decision tree can be called a regression tree.

[0108] Association rules are another technique that can be used in machine learning algorithms. In other words, the machine learning model can be based on one or more association rules. Association rules are created by identifying relationships between variables in large datasets. The machine learning algorithm can identify and / or utilize one or more relationship rules that represent the knowledge derived from the data. The rules can be used, for example, to store, manipulate, or apply this knowledge.

[0109] Machine learning algorithms are typically based on a machine learning model. In other words, the term "machine learning algorithm" can refer to a set of instructions that can be used to create, train, or use a machine learning model. The term "machine learning model" can refer to a data structure and / or a set of rules that represents the learned knowledge (e.g., based on the training performed by the machine learning algorithm). In some implementations, the use of a machine learning algorithm may imply the use of an underlying machine learning model (or multiple underlying machine learning models).

[0110] Machine learning model can imply that the machine learning model and / or the data structure / set of rules that constitute the machine learning model is / are trained by a machine learning algorithm.

[0111] For example, the AI ​​algorithm or machine learning model can be an artificial neural network (ANN). ANNs are systems inspired by biological neural networks, such as those found in a retina or brain. ANNs comprise a plurality of interconnected nodes and a plurality of connections, called edges, between the nodes. There are typically three types of nodes: input nodes, which receive input values; hidden nodes, which are connected (only) to other nodes; and output nodes, which provide output values. Each node can represent an artificial neuron. Each edge can transmit information from one node to another. The output of a node can be defined as a (nonlinear) function of its inputs (e.g., the sum of its inputs).The inputs to a node can be used in the function based on the "weight" of the edge or the node providing the input. The weight of nodes and / or edges can be adjusted during the learning process. In other words, training an artificial neural network can involve adjusting the weights of the nodes and / or edges of the artificial neural network, i.e., to achieve a desired output for a given input.

[0112] Alternatively, the machine learning model can be a support vector machine, a random forest model, or a gradient boosting model. Support vector machines (i.e., support vector networks) are supervised learning models with associated learning algorithms that can be used to analyze data (e.g., in classification or regression analysis). Support vector machines can be trained by providing an input with a plurality of training input values ​​belonging to one of two categories. The support vector machine can be trained to assign a new input value to one of the two categories. Alternatively, the machine learning model can be a Bayesian network, which is a probabilistic directed acyclic graphical model. A Bayesian network can represent a set of random variables and their conditional dependencies using a directed acyclic graph.Alternatively, the machine learning model can be based on a genetic algorithm, which is a search algorithm and heuristic technique that imitates the process of natural selection.

[0113] The term "and / or" encompasses all combinations of one or more of the associated listed elements and can be abbreviated with " / ".

[0114] Although some aspects have been described within the context of a device, it is clear that these aspects also constitute a description of the corresponding process, where a block or device corresponds to a process step or a function of a process step. Similarly, aspects described within the context of a process step also constitute a description of a corresponding block or element or property of a corresponding device.

[0115] Reference sign

[0116] Microscopy system

[0117] Microscopy device (microscope)

[0118] Screen

[0119] Advertisement

[0120] Device control

[0121] Data processing device (computer system) with database

[0122] microscope sample

[0123] detector

[0124] Recording for trial, first point of interest, second point of interest, third point of interest, fourth point of interest

[0125] Step 1: Sample analysis

[0126] Step 2: Saving to internal database

[0127] Step 3: Including an external database

[0128] Step 4: Performing a prediction based on the database

[0129] Step 5: Result in the form of a predicted value

Claims

Patent claims 1. Computer-implemented method for creating a database (105) for an analysis of at least one microscope sample (106) using a microscopy apparatus (101), wherein the method comprises at least the following steps: Receiving input data relating to the detection of at least one action by an operator of the microscopy device (101) in relation to the analysis of the microscope sample (106) using the microscopy device (101), and relating to the microscopy device (101) and / or the microscope sample (106), Generating database data based on the input data, wherein the database data contains at least a relationship between the detected action of the operator and the device (101), the analysis and / or the microscope sample (106), and storing the database data in the database.

2. Method according to claim 1, wherein the relationship comprises a correlation of analysis values ​​relating to the microscope sample (106) and / or spatial areas of image data of the microscope sample (106), at least one system setting of the microscope device (101), a parameter relating to environmental characteristics of the device (101) and / or an analysis result.

3. Method according to claim 1 or 2, wherein the detection of the action is based at least partially on image recognition and / or the detected action relates at least partially to a body part of the operator, optionally an eye, and / or an acoustic signal from the operator.

4. Method according to one of the preceding claims, wherein the detected action of the operator is based on a pupil movement of the operator, in particular the position of the gaze and / or the dwelling of the eye on an area (201 , 202, 203, 204) of the device (101) and / or the opening / closing of at least one eye.

5. Method according to one of the preceding claims, wherein the analysis of the sample (106) extends at least partially to image data and / or analytical values ​​that are displayed using the device (101).

6. Method according to one of the preceding claims, wherein the detected action comprises operating the device (101) by the operator, in particular adjusting an operating element of the device (101) or / or changing the display of the analysis.

7. Method according to one of the preceding claims, wherein input data are excluded from the generation of the database data if they relate to an action of the operator that is excluded from the generation of the database and / or that is unsuitable.

8. Computer-implemented method for determining predictions concerning an analysis of a microscope sample (106) using a microscopy apparatus (101) and a database, comprising Received at least one input relating to the microscope sample (106), analysis and / or microscopy apparatus (101), Creating output data based on the input and the database, optionally created according to a method according to claims 1 to 7, wherein the output data relates to the analysis of the microscope sample (106) using the microscopy device (101) and is based on at least one relationship stored in the database between an action of an operator of the microscopy device (101) and the microscope sample (106), microscopy device (101) and / or analysis, Determine at least one predictive value relating to the analysis, microscope sample (106) and / or microscopy apparatus (101) based on the output data.

9. Method according to claim 8, wherein the predictive value relates to the analysis, forecasting, planning, simulation and / or control of the analysis of the microscope sample (106) using the microscopy device (101).

10. Method according to claim 8 or 9, wherein the predicted value comprises an operating suggestion for the operator to operate the microscopy device (101) for analyzing the microscope sample (106).

11. Method according to any one of the preceding claims 8 to 10, wherein the output data is generated using a computer-based algorithm.

12. Data processing device (105) comprising one or more processors and one or more memories, wherein the processor(s) is / are configured to perform the method according to any one of claims 1 to 7 and / or any one of claims 8 to 11.

13. Computer program product comprising instructions which, when the program is executed by a computer (105), cause the computer to execute the method according to any one of claims 1 to 7 and / or any one of claims 8 to 11.

14. Computer-readable data carrier on which the computer program product according to claim 13 is stored.

15. Method for operating a microscopy apparatus (101) for analyzing at least one microscope sample (106), wherein the method comprises at least the following steps: Providing a microscope sample (106) for analysis using the device (101), Detecting an operator action during the analysis of the microscope sample (106) and identifying the action in relation to the analysis, sample (106) and / or the microscopy device (101), at least partially operating the microscopy device (101) during the analysis of the microscope sample (106) based on the operator action, and at least partially storing data relating to at least one operator action, the analysis, an analysis result, the microscope sample (106) and / or the microscopy device (101) and / or Suggestions for operation and / or evaluation concerning the analysis, the analysis result, the microscope sample (106) and / or the microscopy device (101) to the operator based on the detected action.

16. Method according to claim 15, wherein at least one predictive value relating to the analysis, microscope sample (106) and / or microscopy apparatus (101) is proposed, optionally according to a method according to any one of claims 8 to 11.

17. Method according to one of the preceding claims 15 and 16, wherein the control of the microscopy device and / or an operating suggestion is carried out automatically after a confirming action by the operator.

18. Method according to any one of claims 15 to 17, wherein the microscopy apparatus (101) is operated by virtually actuating a control element (104).

19. Method according to any one of claims 15 to 18, wherein the duration, repetition, frequency and / or amplitude of operator actions is used to identify the action.

20. Method according to any one of claims 15 to 19, wherein detection of the action is based at least partially on image recognition and / or the detected action relates at least partially to a body part of the operator, optionally an eye, and / or an acoustic signal of the operator.

21. Method according to any of the preceding claims 15 to 20, wherein the detected action of the operator is based on a pupil movement of the operator, in particular the position of the gaze and / or the dwelling of the eye on an area (201 , 202, 203, 204) of the device (101) and / or the opening / closing of at least one eye.

22. A method according to any one of the preceding claims 15 to 21, wherein the data are stored, optionally automatically, in a database for the generation of database data, optionally according to a method according to any one of claims 1 to 7.

23. Method according to any one of the preceding claims 15 to 22, wherein data is not stored if it relates to an action of the operator that is excluded from the generation of the database and / or that is unsuitable.

24. Method according to any one of the preceding claims 15 to 23, wherein data is stored which improves the identification of the action of an operator based on the detected action and / or includes feedback on the operator's satisfaction.

25. Microscopy system (100) for analyzing at least one microscope sample (106), wherein the system (100) comprises at least: a recording device (108) for receiving the microscope sample (106), a microscopy device (101) for at least partial analysis of the microscope sample (106) by the operator, a detector (107) for detecting at least one action of the operator relating to the microscopy apparatus (101) and / or the analysis, and one or more units (104, 105) which are / are configured to carry out the method according to any one of claims 15 to 24.

26. Microscopy system (100) according to claim 25, further comprising a data processing device (105) according to claim 12.

Images