Self-learning microscope

Abstract

The present invention relates to a system solution (150) for a microscope (100), a microscope comprising such a system solution and a method of teaching the use of a microscope (100). The system solution (150) comprises: a display device (154); and an input interface (152); the system solution (150) is configured to reproduce a virtual tour using the microscope (100) through the display device in response to user input to the input interface (152).

Description

Technical Field

[0001] The present invention relates to the field of microscopy, and in particular to a system solution for microscopy. The present invention further relates to a method for teaching the use of microscopy. Background Art

[0002] In teaching in schools and universities, the basics of microscopy and the operation of microscopes are currently being taught in order to enable pupils and senior students to acquire images in a relatively high quality manner for later analysis. Time and resources are allocated to secondary goals of teaching. These resources are missing in teaching the actual primary goal of image analysis and, therefore, resources are missing in the focus of the actual teaching content, for example in the fields of biology or medicine, and other fields that require image analysis of microscopic images. In addition, the increasing number of university students puts teachers in schools and universities under increasing time pressure.

[0003] It is therefore a task of the present invention to provide a concept that reduces the amount of resources required to learn microscopy skills. Summary of the invention

[0004] According to the invention, a system solution for a microscope, a microscope comprising such a system solution and a method for teaching the use of a microscope are proposed having the features of the independent claims.

[0005] Advantageous further developments form the subject matter of the dependent claims and of the subsequent description.

[0006] The present invention solves this task by providing individual guidance to the user of the microscope using the necessary steps for image optimization. Thus, through direct interaction with the microscope, the intervention of a teacher is no longer required to learn a basic understanding of the working principles of the microscope. This helps to focus attention on the sample to be examined.

[0007] In an embodiment, the present invention relates to a system solution (or user arrangement) for, i.e. for use with, a microscope. The system solution comprises a display device and an input interface. The system solution (e.g., one or more processors or processing devices thereof) is configured (or designed) to reproduce (or play or provide) a virtual tour of the microscope (and its use) via the display device in response to user input to the input interface. The display device may, for example, be connected to an image sensor for displaying microscope images and may also be connected to a network, such as a university network for displaying course content.

[0008] The display device may, for example, include one or more displays. Preferably, the display device is configured to display (or superimpose) additional information of the virtual tour into a real real-time image (an image actually seen or generated by the user, such as an image of a sample) or superimpose it (i.e., the additional information) onto a real real-time image by augmented reality. This can be achieved by combining text with a transparent background with a real-time image. Such a display may be a display separated from the microscope or a display attached to the microscope, such as in the eyepiece of the microscope. In addition, the display device may also include a hybrid eyepiece, in which the additional information of the virtual tour is superimposed or displayed on the real real-time image by augmented reality. The input interface may have, for example, one or more keys, such as in the form of a real key or a touch screen. Alternatively, voice control or gesture control, etc. may be used as an input interface. After the user input to start the virtual tour, for example, by pressing a button or by voice or gesture input, the stored tour is started by operating the microscope, and in particular, the user is guided to acquire at least one image so as to obtain the best image result through step-by-step guidance. Preferably, the optimal sample image is eventually superimposed on the user-generated image again and again (or repeatedly) until a match is achieved (ie, until the optimal sample image and the true real-time image match each other).

[0009] Preferably, the system solution is configured (or adapted) to generate feedback to the user of the system solution, particularly in the event that the user deviates from a predetermined sequence of system settings that should have been completed (or performed) by the user (e.g., through the aforementioned step-by-step instructions). This can be achieved by embedding sensors at user interface points that will detect when a position is moved outside of the specification (e.g., a specified position) or moved in a sequence outside of the specification. Feedback can be in the form of at least one of the following: audio (e.g., voice, beep), video, flashing light, electric shock to the user, and tactile feedback.

[0010] Advantageously, the system solution is configured to provide a certificate to a user in a test environment (competition for completing a task), such as a student class, such as a gift won or a test passed certificate. Each participant has access to a microscope network (including a server) as well as a microscope and the system solution. The provision of a certificate can be in the form of an image pop-up or a musical sound or a credit that is deposited into a student's course account.

[0011] It is advantageous if the system solution is able to adapt operating components (components or parts for operating the system solution or the microscope, such as the focus drive) according to the virtual tour, in particular in a predetermined sequence of steps or in a way that makes learning easier. In other words, the system solution can be configured to adapt the operating components (and the way they are operated) according to the virtual tour. For example, in this way it may be easier for the user to focus upwards rather than downwards, if this is what the user should do according to the virtual tour (the same applies to magnification, condenser adjustment, sample positioning, eyepiece adjustment, etc.). This can be achieved by using electromagnets, which generate a stronger or weaker torque depending on the position of the operating component.

[0012] Preferably, the system solution is configured to allow updating or exchanging (e.g. by a teacher) virtual tours, in particular regarding teaching sequences or teaching procedures, or the teacher can generate new teaching templates. The teacher can then access, for example, the course templates on the university network, from which he can create sequences to match his teaching topics. The student can then access his laboratory manual from the network and the microscope procedures will be updated.

[0013] Advantageously, the system solution is configured to bring the system solution and / or the microscope into a predetermined situation, in particular at startup or shutdown. Default positions of movable settings (e.g. magnification, focus, condenser) may be programmable, e.g. by a teacher, so that the system remembers where to start a session.

[0014] It is advantageous if the system solution is configured to provide (or determine) quality information about the use or operation of the system solution and / or the microscope (e.g. by means of an evaluation device), such as the quality level of the operation or the quality level of the images acquired and / or the results to be achieved using the system (system solution and / or the microscope). In particular, the evaluation device is further configured to output the quality level or quality information, e.g. by means of a display device. The recipient of this output may be a user and / or a teacher. This may be done by means of a "knowledge check" or an exam, which may be designed by the teacher and which the student may access from, e.g., a university network.

[0015] Preferably, the quality information is determined by machine learning, deep learning and / or pattern recognition, etc. The system solution can then be embedded with appropriate imaging software.

[0016] In another embodiment, the invention also relates to a microscope (or microscope system) having (or comprising) a system solution or a user arrangement as described above. According to this aspect, the system solution or its individual components can be embedded in a microscope (or microscope system).

[0017] In a further embodiment, a standard microscope can be equipped with a novel eyepiece (for augmented reality) and a control panel (input interface), for example 4-5 buttons can be attached to the side of the microscope base to achieve the above functions.

[0018] In further embodiments, the invention also relates to a method (or process) and a computer program for using the above-mentioned system solution or user arrangement.

[0019] Further advantages and embodiments of the invention will become apparent from the description and drawings.

[0020] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0021] Although some aspects have been described in the context of an apparatus or system, it is clear that these aspects also represent a description of the corresponding method, where the blocks or devices correspond to method steps or features of method steps. Similarly, aspects described in the context of method steps also represent a description of the corresponding blocks or items or features of the corresponding apparatus.

[0022] It should be pointed out that the features mentioned above and the features yet to be described further below can be used not only in the respectively indicated combination but also in further combination or alone, without departing from the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A microscope according to the invention is schematically shown in a preferred embodiment.

[0024] Figure 2A and 2B The components of the system solution according to the invention are schematically shown in different preferred embodiments. DETAILED DESCRIPTION

[0025] exist Figure 1 , a microscope 100 according to the invention in a preferred embodiment is schematically shown. The microscope 100 comprises a stand or base 102 with a stage 104 movable in the vertical or z direction by a focus drive 112, an objective 108 and an eyepiece 110 (optically connected to the objective 108, for example via a lens tube). On the stage, an object or sample 106 to be observed or examined by the microscope 100 is placed.

[0026] Furthermore, in a preferred embodiment, a system solution 150 according to the invention is provided for the microscope 100. For example, the system solution 150 comprises a display device 154 in the form of a display or monitor, an input interface 152 in the form of a keyboard and a processing device 156, such as a PC, for performing the necessary processing steps.

[0027] The system solution 150 is configured to reproduce a virtual tour using the microscope 100 through the display device 154 in response to user input to the input interface 152. In addition, for example, the system solution (150) includes an image sensor 158 attached to the microscope 100 so as to acquire an image seen by the user of the microscope 100 via the eyepiece and provide the image in the display device.

[0028] exist Figure 2A In the embodiment, a display device 254 is shown as part of the system solution according to the present invention in a preferred embodiment. The display device 254 may correspond to or be used as Figure 1 The display device 154 shown in FIG.

[0029] For example, using a microscope, e.g. Figure 1 The virtual tour of the microscope 100 shown in FIG. 1 is shown with two steps 260 and 262 guiding the user to the best image of the sample to be viewed.

[0030] In a first step 260, instructions may be displayed to move the sample in a certain direction, for example, to the right. In addition, the actions or steps to be performed by the user may be as follows: Figure 2A As shown in , it is visually shown next to block 260.

[0031] In a second step 262, it can be shown that the focus drive is operated (rotated) in a certain direction (see Figure 1 ) instructions. In addition, the actions or steps to be performed by the user can be as follows Figure 2A 262. Preferably, the focus drive can be adjusted (eg, by using an electromagnet) so that the direction of rotation that the user wants to use is easier to perform than another (wrong) direction.

[0032] exist Figure 2B , a display device 210 is shown as part of a system solution according to the invention in a further preferred embodiment. For example, the display device 210 comprises a hybrid eyepiece (or, alternatively, is integrated into the eyepiece). Such a hybrid eyepiece can be used to replace Figure 1 210 of the microscope shown in . In this case, the (hybrid) eyepiece is part of the system solution and (e.g. after attaching or replacing the original eyepiece) the microscope (which itself may include the system solution).

[0033] Similar to Figure 2A Instructions for the virtual tour shown in , instructions or other additional information may be displayed in a display device or hybrid eyepiece and thus may be superimposed or added to the actual real-time image 206 of the sample to be observed via augmented reality.

[0034] For example, instructions in the form of arrows (up and right) are displayed next to the true real-time image 206 to teach the user to move the sample that should match the displayed, for example, optimal sample image 206 ′ in the display device 210 or hybrid eyepiece.

[0035] Some embodiments relate to a microscope comprising: Figures 1 to 2B Alternatively, the microscope may be combined with Figures 1 to 2B A portion of or connected to the system described in one or more of the accompanying drawings. Figure 1 A schematic diagram of a system 150 configured to perform the methods described herein is shown. The system 150 can be part of the microscope 100 or embedded in the microscope 100 and can include a computer system 156. The microscope 100 is configured to capture images and is connected to the computer system 156. The computer system 156 is configured to perform at least a portion of the methods described herein. The computer system 156 can be configured to execute a machine learning algorithm. The computer system 156 and the microscope 100 can be separate entities, but can also be integrated together in a common housing. The computer system 156 can be part of a central processing system of the microscope 100 and / or the computer system 156 can be part of a subcomponent of the microscope 100, such as a sensor, actuator, camera or lighting unit of the microscope 100, etc.

[0036] The computer system 156 may be a local computer device (e.g., a personal computer, a laptop, a tablet computer, or a mobile phone) having one or more processors and one or more storage devices, or may be a distributed computer system (e.g., a cloud computing system having one or more processors and one or more storage devices distributed in various locations, for example, at a local client and / or one or more remote server clusters and / or data centers). The computer system 156 may include any circuit or combination of circuits. In one embodiment, the computer system 156 may include one or more processors that may be of any type. As used herein, a processor may refer to any type of computing circuit, such as, but not limited to, for example, a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA), or any other type of processor or processing circuit. Other types of circuits that may be included in the computer system X20 may be custom circuits, application specific integrated circuits (ASICs), etc., such as, for example, one or more circuits (such as communication circuits) for wireless devices, such as mobile phones, tablet computers, laptops, two-way radios, and similar electronic systems. The computer system X20 may include one or more storage devices, which may include one or more memory elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard disk drives and / or one or more drives for handling removable media, such as compact disks (CDs), flash memory cards, digital video disks (DVDs), etc. The computer system X20 may also include a display device, one or more speakers, and a keyboard and / or controller, which may include a mouse, a trackball, a touch screen, a voice recognition device, or any other device that allows a system user to input information to and receive information from the computer system X20.

[0037] Some or all method steps can be performed by (or using) hardware devices, such as processors, microprocessors, programmable computers or electronic circuits. In some embodiments, some of the most important method steps can be performed by such devices.

[0038] According to certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. The implementation may be performed using a non-transitory storage medium with electronically readable control signals stored thereon, such as a digital storage medium, for example a floppy disk, DVD, Blu-ray, CD, ROM, PROM and EPROM, EEPROM or FLASH memory, which cooperates (or can cooperate) with a programmable computer system to perform the corresponding method. Therefore, the digital storage medium may be computer readable.

[0039] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that the system performs one of the methods described herein.

[0040] Generally, embodiments of the present invention can be implemented as a computer program product with a program code, when the computer program product runs on a computer, the program code is operative to perform one of the methods.For example, the program code can be stored on a machine readable carrier.

[0041] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0042] In other words, an embodiment of the inventive method is, therefore, a computer program with a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0043] A further embodiment of the present invention is therefore a storage medium (or a data carrier, or a computer-readable medium) comprising a computer program stored thereon for performing one of the methods described herein when executed by a processor. The data carrier, the digital storage medium or the recorded medium is typically tangible and / or non-transitory. A further embodiment of the present invention is an apparatus as described herein, comprising a processor and the storage medium.

[0044] A further embodiment of the invention is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.The data stream or the sequence of signals may, for example, be configured to be transmitted via a data communication connection, for example via the Internet.

[0045] A further embodiment comprises a processing means, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0046] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0047] A further embodiment according to the invention comprises an apparatus or a system configured to transmit (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. For example, the receiver may be a computer, a mobile device, a memory device, etc. For example, the apparatus or the system may comprise a file server for transmitting the computer program to the receiver.

[0048] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may collaborate with a microprocessor to perform one of the methods described herein. Typically, these methods are preferably performed by any hardware device.

[0049] Embodiments may be based on the use of machine learning models or machine learning algorithms. Machine learning may refer to algorithms and statistical models that a computer system can use to perform specific tasks, without explicit instructions, but relying on models and reasoning. For example, in machine learning, data transformations inferred from analysis of historical and / or training data may be used instead of rule-based data transformations. For example, a machine learning model may be used or a machine learning algorithm may be used to analyze the content of an image. In order for a machine learning model to analyze the content of an image, a machine learning model may be trained using training images as input and training content information as output. By training a machine learning model using a large number of training images and / or training sequences (e.g., words or sentences) and associated training content information (e.g., labels or annotations), the machine learning model "learns" to recognize the content of the image, so the machine learning model may be used to recognize image content that is not included in the training data. The same principle may also be used for other types of sensor data: by training a machine learning model using training sensor data and a desired output, the machine learning model "learns" the transformation between sensor data and output, which can be used to provide output based on non-training sensor data provided to the machine learning model. The provided data (e.g., sensor data, metadata, and / or image data) may be preprocessed to obtain a feature vector, which is used as input to the machine learning model.

[0050] The machine learning model can be trained using training input data. The example specified above uses a training method called "supervised learning". In supervised learning, the machine learning model is trained using multiple training samples, each of which may include multiple input data values ​​and multiple expected output values, i.e., each training sample is associated with an expected output value. By specifying the training samples and the expected output values, the machine learning model "learns" which output value is provided based on input samples similar to the samples provided during training. In addition to supervised learning, semi-supervised learning can also be used. In semi-supervised learning, some training samples lack corresponding expected output values. Supervised learning can be based on supervised learning algorithms (such as classification algorithms, regression algorithms, or similarity learning algorithms). When the output is limited to a limited set of values ​​(categorical variables), a classification algorithm can be used, i.e., the input is classified into one of a limited set of values. When the output can have any numerical value (within a range), a regression algorithm can be used. Similarity learning algorithms may be similar to classification and regression algorithms, but are based on learning from examples using a similarity function that measures the similarity or correlation between two objects. In addition to supervised or semi-supervised learning, unsupervised learning can be used to train machine learning models. In unsupervised learning, one may be provided with (only) input data, and an unsupervised learning algorithm may be used to find structure in the input data (e.g., by grouping or clustering the input data, finding commonalities in the data). Clustering is the assignment of input data, including multiple input values, into subsets (clusters) based on one or more (predefined) similarity criteria, so that input values ​​within the same cluster are similar, while being different from input values ​​included in other clusters.

[0051] Reinforcement learning is the third group of machine learning algorithms. In other words, reinforcement learning can be used to train machine learning models. In reinforcement learning, one or more software actors (called "software agents") are trained to take actions in an environment. Based on the actions taken, rewards are calculated. Reinforcement learning is based on training one or more software agents to select actions such that the cumulative reward increases, thereby causing the software agent to become better at completing a given task (as evidenced by increasing rewards).

[0052] In addition, some techniques can be applied to some machine learning algorithms. For example, feature learning can be used. In other words, a machine learning model can be trained at least in part using feature learning, and / or a machine learning algorithm can include a feature learning component. Feature learning algorithms, also called representation learning algorithms, can preserve the information in the input but can also transform it in a way that makes it useful, usually as a preprocessing step before performing classification or prediction. For example, feature learning can be based on principal component analysis or cluster analysis.

[0053] In some examples, anomaly detection (i.e., outlier detection) can be used, the purpose of which is to provide identification of input values ​​that raise suspicion by being significantly different from the majority of inputs or training data. In other words, a machine learning model can be trained at least in part using anomaly detection, and / or a machine learning algorithm can include an anomaly detection component.

[0054] In some examples, a machine learning algorithm may use a decision tree as a prediction model. In other words, a machine learning model may be based on a decision tree. In a decision tree, an observation about an item (e.g., a set of input values) may be represented by a branch of the decision tree, and an output value corresponding to the item may be represented by a leaf of the decision tree. A decision tree may support both discrete and continuous values ​​as output values. If discrete values ​​are used, a decision tree may be represented as a classification tree, and if continuous values ​​are used, a decision tree may be represented as a regression tree.

[0055] Association rules are a further technique that can be used with machine learning algorithms. In other words, a machine learning model can be based on one or more association rules. Association rules are created by identifying relationships between variables in a large amount of data. A machine learning algorithm can identify and / or utilize one or more relationship rules that represent knowledge derived from the data. Rules can, for example, be used to store, manipulate, or apply knowledge.

[0056] Machine learning algorithms are often based on machine learning models. In other words, the term "machine learning algorithm" may refer to a set of instructions that can be used to create, train, or use a machine learning model. The term "machine learning model" may refer to a data structure and / or a set of rules that represent learned knowledge (e.g., based on training performed by a machine learning algorithm). In embodiments, the use of a machine learning algorithm may imply the use of an underlying machine learning model (or multiple underlying machine learning models). The use of a machine learning model may mean that the machine learning model and / or the data structure / rule set that is the machine learning model is trained by a machine learning algorithm.

[0057] For example, a machine learning model may be an artificial neural network (ANN). ANN is a system inspired by biological neural networks, such as those found in the retina or brain. ANN comprises a plurality of interconnected nodes and a plurality of connections between the nodes, so-called edges. There are generally three types of nodes, input nodes that receive input values, hidden nodes that are (only) connected to other nodes, and output nodes that provide output values. Each node may represent an artificial neuron. Each edge may transfer information from one node to another. The output of a node may be defined as a (non-linear) function of its input (e.g. the sum of its inputs). The input of a node may be used in a function based on the "weight" of the node or edge that provides the input. The weights of the nodes and / or edges may be adjusted during the learning process. In other words, the training of an artificial neural network may include adjusting the weights of the nodes and / or edges of the artificial neural network, i.e. achieving the desired output for a given input.

[0058] Alternatively, the machine learning model can be a support vector machine, a random forest model, or a gradient boosting model. A support vector machine (i.e., a support vector network) is a supervised learning model with an associated learning algorithm that can be used to analyze data (e.g., in classification or regression analysis). A support vector machine can be trained by providing multiple training input values ​​belonging to one of two categories to the input. A support vector machine can be trained to assign new input values ​​to one of two categories. Alternatively, the machine learning model can be a Bayesian network, which is a probabilistic directed acyclic graph model. A Bayesian network can represent a set of random variables and their conditional dependencies using a directed acyclic graph. Alternatively, the machine learning model can be based on a genetic algorithm, which is a search algorithm and heuristic technique that mimics the natural selection process.

Claims

1. A system (150) for a microscope (100), comprising: A display device (154, 254, 210), wherein the display device (154, 254) includes one or more displays; as well as Input interface (152); The system (150) is configured to reproduce a virtual tour using the microscope (100) through the display device (154, 254, 210) in response to a user input to the input interface (152), wherein the virtual tour is displayed on the display and guides the user step by step to obtain an optimal image of the sample to be observed by displaying arrows next to the real real-time image of the sample on the display, the arrows indicating the direction to move the sample or the direction to adjust the operating components of the microscope, wherein the optimal image is superimposed with the real-time image of the image generated by the user until the optimal image and the real-time image match each other.

2. The system (150) of claim 1, wherein: The display device (154, 254) is configured to display the additional information of the virtual tour in the real real-time image (206) through augmented reality, or to superimpose the additional information of the virtual tour on the real real-time image (206).

3. The system (150) according to claim 1, configured to generate feedback to a user of the system (150) in case the user deviates from a predetermined sequence of system settings that should be completed by the user.

4. The system (150) of claim 3, wherein: The feedback may be in the form of at least one of: audio, video, a flashing light, an electric shock to the user, and tactile feedback.

5. The system (150) of claim 1, configured to provide a certificate to a user in a testing environment, wherein the certificate indicates a gift won or a test passed.

6. The system (150) of claim 1, the system being configured to make it more difficult for a user to adjust the operating component (112) in a direction opposite to the direction than for the user to adjust the operating component in the direction.

7. The system (150) of claim 1, configured to allow updating or exchanging the virtual tour.

8. The system (150) according to claim 1, configured to cause the system (150) and / or the microscope (100) to enter a predetermined situation.

9. The system (150) according to claim 1, configured to provide quality information about the use of the system (150) and / or the microscope (100), and to output the quality information.

10. The system (150) according to claim 9, wherein the quality information is determined by machine learning, deep learning and / or pattern recognition.

11. The system (150) according to claim 1, wherein the display device is integrated in an eyepiece (110) of the microscope (100).

12. The system (150) of claim 1, wherein the display device comprises a hybrid eyepiece (210), the hybrid eyepiece (210) being configured to display additional information of the virtual tour in the real real-time image (206) through augmented reality, or to superimpose the additional information of the virtual tour on the real real-time image (206).

13. A microscope (100) comprising a system (150) according to any one of the preceding claims.

14. A method for teaching the use of a microscope (100), in particular by means of a system (150) according to any one of claims 1 to 12, wherein: In response to user input to the input interface (152), a virtual tour using the microscope (100) is displayed via a display device (154, 254, 210).

15. A computer-readable storage medium comprising a computer program with a program code for executing the method according to claim 14 when the computer program is run on a processor (156), in particular on a processor (150) of a system (150) of a microscope (100).

Images