Method, system, light microscope and computer program for processing light-microscopy data sets

Abstract

The invention relates to a method for processing light-microscopy data comprising: providing predefined positions (11) on a user interface (10), inserting at least one data symbol (13), which represents a light-microscopy data set (20), at one of the predefined positions (11) on the user interface (10), inserting at least one operator symbol (14), which represents an operation which was or is to be carried out on the light-microscopy data set (20), at one of the predefined positions (11) on the user interface (10), and applying the operation represented by the operator symbol (14) to the light-microscopy data set (20) in a processing sequence which is defined by an arrangement of the at least one data symbol (13) and the at least one operator symbol (14) on the user interface (10) along a predefined processing direction (V). The invention also relates to a system comprising a display unit (8) and a computing unit (9), to a light microscope (1) and to a computer program for carrying out the method.

Description

[0001] Method, system, light microscope and computer program for processing light microscope data sets

[0002] Technical field of the invention

[0003] The invention relates to a method for processing light microscopic data sets by arranging data symbols and operator symbols on a graphical user interface, as well as a system comprising a computing unit and a display unit, a light microscope with the system according to the invention and a computer program for carrying out the method.

[0004] State of the art

[0005] State-of-the-art digital data processing software allows data, such as image data, and operations, such as filters, used to process the image data to be displayed in a structured layered manner on a graphical user interface. When applying multiple operations, a processing sequence can be maintained that corresponds to the layered representation. Furthermore, new data or operators can be inserted through user interaction, and the order of the stack and parameters of the operations can be changed.

[0006] Furthermore, special programs are known for processing light microscopic data in particular. Data sets and operations are represented by symbols on a user interface. Data symbols representing a particular data set can be linked, for example, via connecting lines, to operator symbols representing arithmetic operations to perform the operation on the data set. Parameters of the arithmetic operations can be adjusted, for example, via input menus. Such arrangements of data symbols and operator symbols are also known as filter graphs.

[0007] According to the current state of the art, a user can freely arrange a large number of data symbols and operator symbols on the user interface. This enables the flexible execution of even complex processing routines, but has the disadvantage that the display quickly becomes confusing for the user. Furthermore, the extensive freedom of arrangement on the user interface tempts inexperienced users in particular to create filter graphs that lead to suboptimal or erroneous processing sequences.

[0008] Object of the invention

[0009] Therefore, it is the object of the present invention to provide a method for processing light microscopic data sets that offers the user a clear, robust and error-free graphical representation of the data sets and processing steps, thus improving user-friendliness and increasing the effectiveness of data processing.

[0010] Solution

[0011] This object is achieved by the subject matter of independent claims 1 (method), 23 (system), 25 (light microscope), and 26 (computer program). Advantageous developments of the invention are specified in subclaims 2 to 22 and 24 and are described below.

[0012] Description of the invention

[0013] A first aspect of the invention relates to a method for processing light microscopic data, comprising the steps of: providing predefined positions on a two-dimensional user interface displayed by a display unit; inserting at least one data symbol representing a light microscopic data set at one of the predefined positions on the user interface; inserting at least one operator symbol representing an operation to be performed on the light microscopic data set at at least one of the predefined positions on the user interface;and applying the operation or operations represented by the at least one operator symbol to the at least one light microscopic data set represented by the data symbol or data symbols in a processing order determined by an arrangement of the at least one data symbol and the at least one operator symbol on the user interface along a predetermined processing direction;

[0014] The at least one data symbol and the at least one operator symbol form a graphical representation of an ordered sequence of operations (processing steps) on one or more light microscopic data sets, which can also be referred to as a processing pipeline.

[0015] The predefined positions, which are distributed in two dimensions, particularly regularly, across the user interface, define possible positions for the data symbols and operator symbols. Combined with the predefined processing direction, which reflects the desired processing sequence of operations, this results in a clearly organized and clear graphical representation and, in particular, allows the user to only link specific data and operations (for example, loops and circular arguments can be excluded). This makes the processing of light microscopy data more effective, less error-prone, and more user-friendly.

[0016] The steps of the method do not necessarily have to be carried out in the specified order. For example, it is possible for the predefined positions to be made available only after a data symbol or an operator symbol has been inserted. What is crucial is not their absolute positions, but the relative arrangement between the data symbols and operator symbols. A zoom operation, in which the distances between all predefined positions are scaled, is therefore also possible within the scope of the method according to the invention. The fact that the steps do not have to be carried out in the specified order also means that, for example, an operator symbol can be arranged first and then a data symbol on the user interface.

[0017] In particular, the processing direction may run along several predefined positions arranged in a line, but the corresponding predefined positions may also be positioned along a non-linear path on the user interface, as long as the processing direction is clearly recognizable for the user.

[0018] The data symbols and operator symbols can be displayed on the user interface, for example, as circles or rectangles, particularly colored ones. The shape and / or color of the symbols can be different for data symbols and operator symbols (and optionally for other symbols described below) to make them easier to distinguish. Whenever this specification refers to data symbols, operator symbols and / or other symbols being arranged at a predefined position, this means in particular that a defined point of the symbol (in the case of rectangles, for example, a center point or a specific corner, e.g., the top left corner) is placed at the predefined position.

[0019] The data symbols and operator symbols can be placed at the predefined positions on the user interface, in particular by user interaction, such as drag-and-drop using a computer mouse or by swiping on a touch-sensitive screen. It is also conceivable, for example, that a data symbol is generated by activating a button on an image shown by the display unit or executing a menu command, or by dragging the image onto the user interface using a mouse or touchscreen interaction. In addition, it is also conceivable that at least certain operator symbols can be dragged from the user interface onto displayed images in order to execute the operation underlying the operator symbol on the image. Certain data symbols and / or operator symbols can also be arranged at the predefined positions automatically, i.e. without user interaction. This can, for example,be useful for a data symbol if the corresponding light microscopic data set represented by the data symbol is generated by components of a light microscope coupled to the display unit, or if the data set is loaded from an external source. The predefined positions can optionally be displayed on the user interface, e.g. as dots or crosses or as outlines in the size and shape of a data symbol and / or an operator symbol. Alternatively, the predefined positions are not displayed; for example, when a data symbol or operator symbol is moved across the user interface, the respective symbol is highlighted in color, enlarged or otherwise marked when the data symbol or operator symbol is at the predefined position.Another or additional possibility is for the data or operator symbol to be placed to "stick" to the predefined positions when moving across the user interface induced by user interaction, i.e., to react only with a delay to user interaction. It is also conceivable to move the symbol in discrete steps along the predefined positions in response to a user input intended to cause the symbol to move.

[0020] A light microscopy dataset can be or include, for example, a light microscopy image, a sequence of light microscopy images (e.g., an axial stack of images of different focal planes or a temporal sequence of images), raw data, such as localization data of an emitter (which, for example, in the case of MINFLUX localization, can include sampling positions and associated photon counts and / or lists of events), a localization map of multiple emitters in a sample, or a trajectory representing the movement of one or more emitters in the sample. A region of a sample can also be a light microscopy dataset. This can, for example, be stored as a list of coordinates of an image, in particular the coordinates that uniquely identify the outline of the region (e.g., corners of a rectangle or center and radius of a circle), or the coordinates of all pixels that form the region.The light microscopic data set can in particular also consist of several sub-data sets, which can be represented, for example, in the form of a vector of images or a matrix of pixels. Such sub-data sets can be or include, for example, different color channels of an image, different regions of interest of a sample, or different objects detected (automatically or manually) in an image of the sample. Of course, color channels, regions of interest or objects can also be saved and processed as separate light microscopic data sets. If several sub-data sets are summarized in a data symbol, these sub-data sets can in particular be processed together with an operation that is represented by an operator symbol. Then, in particular, several processed images can also be processed automatically or in response to a user input, e.g.placing an output icon at the appropriate position on the user interface.

[0021] Localization data consists, in particular, of positions calculated for an individual emitter based on detected light emissions. The light microscopic data set can be acquired using a light microscope coupled to the display unit and subsequently processed using the method according to the invention. However, it is also possible to save light microscopic data sets acquired at an earlier time and, in particular, at a different location and then process them using the method. The light microscopic data set can be generated using various light microscopic techniques, e.g., scanning fluorescence microscopy, wide-field fluorescence microscopy, localization microscopy (especially MINFLUX microscopy), or super-resolution microscopy (especially STED microscopy).

[0022] According to one embodiment, the at least one data symbol has a data output, which is particularly designed to transfer the light microscopic data represented by the data symbol to an operator, wherein the operation performed by the operator is represented by an operator symbol. A data output is particularly assigned to a position of the data symbol, wherein furthermore in particular the data is transferred to an operator when the operator symbol representing the corresponding operation, in particular a data input of the operator symbol, is arranged at the data output. In particular, the data output is arranged at a position that lies in the processing direction from the predefined position of the data symbol. Data inputs can be provided in particular for certain types of light microscopic data sets.In particular, an operator symbol can also have data inputs for different types of light microscopy data sets. For example, a so-called crop operator could be represented by an operator symbol that has one data input for an image data set and at least one additional data input for region data. The underlying operator could then, for example, crop the region specified by the region data from the image. If the region data specifies multiple regions, the operator symbol could also have multiple data outputs, each of which outputs a partial image corresponding to a respective region.

[0023] The data passed to an operator at data inputs does not necessarily have to originate from a light microscope data set represented by a data symbol. For example, data from a component of a light microscope or a component connected to a light microscope (e.g., a pump for controlling the sample medium) can also be read at a data input. The data may contain, for example, information about the state of the component or a measured value measured by the component. In this way, an operation represented by an operator symbol can, for example, take the state of a component into account.

[0024] According to a further embodiment, the at least one operator symbol has at least one data input and at least one data output. The data input or each data input is designed in particular to transfer light microscopic data represented by a data symbol or processed data from a further operator symbol or another symbol to an operator, wherein the operation performed by the operator is represented by the operator symbol. The data output or each data output of the operator symbol is designed in particular to transfer data processed by means of the operation represented by the operator symbol. A data output is assigned in particular to a position of the operator symbol, wherein furthermore in particular the data is transferred when the corresponding data symbol or further operator symbol is arranged at the data input or data output.In particular, the data input is arranged at a position that lies opposite to the processing direction from the predefined position of the operator symbol. In particular, the data output is arranged at a position that lies in the processing direction from the predefined position of the operator symbol.

[0025] According to a further embodiment, the at least one data input and / or the at least one data output is graphically identified. The data inputs and / or data outputs can be marked, for example, in color or with symbols, in particular differently for data inputs and data outputs. The data inputs and / or data outputs can always be highlighted graphically, or only when they are linked to other symbols on the user interface. In the latter case, the data inputs and / or outputs can also be represented, for example, by lines that connect symbols on the user interface. The operator represented by the operator symbol can process one light microscopic data set or multiple light microscopic data sets. In the case of multiple data inputs, the data sets are in particular offset against one another, but can also be processed in parallel.A data input is in particular a position of an operator symbol, adjacent to which a data symbol can be arranged in order to process the data set with the operation. This position can, but does not necessarily have to, be highlighted graphically. If two or more data inputs are provided, the operator symbol can, in particular, be wider than the data symbols so that a respective data symbol can be arranged at each data input. In the event that multiple data outputs are provided, this means in particular that two processed data sets are output as a result of the processing operation. An example of an application for this is the spectral separation of a light microscope image into multiple color channels, with the color channels being output at the data outputs.

[0026] According to a further embodiment, an arrangement of an operator object and / or a data object at a predefined position of the user interface is only permitted if the operator object or the data object has a data input that is compatible with a data output of an operator object or data object arranged at an adjacent predefined position, or if the operator object or the data object has a data output that is compatible with a data input of an operator object or data object arranged at an adjacent predefined position.

[0027] According to a further embodiment, parameters of the processing operation represented by the operator symbol are assigned to the operator symbol. In particular, the parameters can be changed by user input, e.g., by opening a dropdown menu with at least one input field for entering the parameters. Such parameters can, among other things, also contain conditions under which the operation represented by the operator symbol is executed. In this way, IF loops can be implemented, particularly in combination with parallel processing threads. In particular, an operator symbol whose parameters have been changed can be saved as a user-defined operator symbol and reused later.

[0028] According to one embodiment, the predefined positions form a regular two-dimensional grid. The grid can be, for example, a Cartesian grid or a hexagonal grid. On a regular two-dimensional grid, the processing direction is particularly easy to recognize and intuitive for the user.

[0029] According to a further embodiment, the processing direction on the user interface is vertical. This is a particularly simple and intuitive arrangement for the user, as it corresponds to a stack of operations. In this case, for example, a data symbol can be arranged at the top end of a chain of symbols on the user interface, with the chain running from top to bottom in the vertical direction. The chain can, for example, comprise a data symbol and a plurality of operator symbols, with the arithmetic operations underlying the operator symbols being carried out in the order of the operator symbols in the processing direction. Of course, the reverse case is also possible, namely that a data symbol is arranged at the bottom end of a vertical chain and the processing direction runs from bottom to top.In the horizontal direction, several processing threads of symbols can then be arranged next to each other in order to carry out parallel operations or to implement links.

[0030] According to a further embodiment, the processing direction is horizontal. This representation is also clear and intuitive, as it can particularly correspond to the reading direction of texts. For example, a data symbol can be placed at the left end of a chain that includes several operator symbols in addition to the data symbol. The processing direction then runs, in particular, from left to right. The reverse case (a direction from right to left) is also possible. Processing threads for parallel operations or links can then be arranged one above the other, particularly in the vertical direction.

[0031] According to a further embodiment, the operations are selected from the group: basic arithmetic operations, intensity scaling, linear image deconvolution, iterative image deconvolution, noise reduction / removal, thresholding, masking, image segmentation, demixing, automatic object detection, resampling, color adjustment, brightness adjustment, adjustment of a display area, coordinate transformation.

[0032] The basic arithmetic operations include addition, subtraction, multiplication, and division. Addition is useful, for example, for overlaying multiple color channels in a combined image. Subtraction can be used, for example, to subtract background or baseline intensity from a light microscopy data set. Multiplication and division allow, for example, the normalization of image data and / or a weighted overlay of multiple data sets.

[0033] In intensity scaling, for example, multiplies the intensities of all pixels in an image by a factor (which can be smaller or larger than one).

[0034] Linear or iterative image deconvolution can particularly improve the resolution of an image.

[0035] In particular, noise reduction involves applying filters to image data to reduce noise.

[0036] A thresholding operation specifically involves comparing pixel intensities with a threshold. If the threshold is reached or exceeded, the pixel value can be set to 1, for example; if the threshold is exceeded, the value can be set to 0. The result of such an operation is a binary mask used to capture specific sample structures. Such binary masks can then serve as input data for further operations, such as image segmentation.

[0037] In particular, masking involves applying a mask to an image, meaning that certain areas of the image are transferred to a new image, while other areas are not.

[0038] Image segmentation involves dividing a light microscopic image into at least two regions. For example, a region of interest (ROI) can be selected from an overview image.

[0039] Image segmentation or automatic object recognition can be carried out in particular by means of an artificial intelligence algorithm, further in particular by means of a deep learning algorithm, e.g. by means of an artificial neural network.

[0040] Partial images obtained during image segmentation or object recognition can in particular be stored in separate light microscopic data sets or as sub-data sets of a higher-level light microscopic data set represented by the same data symbol.

[0041] In the context of this application, the term unmixing refers to the separation of different spectral channels or lifetime channels of a data set.

[0042] In particular, resampling involves bringing two images to the same number of pixels by combining neighboring pixels for one of the images (e.g., by averaging their intensities) or by dividing each pixel for one of the images into several pixels by interpolation.

[0043] For example, color adjustment can apply a new color scale to image data.

[0044] A brightness adjustment includes, for example, a gamma correction.

[0045] Adjusting the display area includes zooming, as well as so-called crop and pan operations.

[0046] During a coordinate transformation, for example, a new image can be created based on the coordinate transformation or image areas can be combined.

[0047] According to a further embodiment, the operator symbol represents a combination of several operations, in particular wherein the operations are applied to the at least one light microscopic data set by calling a script.

[0048] According to a further embodiment, the method comprises generating a data output based on the applied operation or operations. According to a further embodiment, the data output comprises displaying at least one processed image, diagram, geometric shape, text, or at least one number, or saving or exporting a processed data set.

[0049] A diagram can be, for example, a histogram or a phasor plot. Text output can be combined with image data as annotation, e.g., by marking detected objects in an image with an arrow and a label or number. Geometric shapes can also be combined with image output, e.g., by drawing a box around a detected region of interest. Processed images can be saved or exported as image data; lists and other texts can be exported, for example, in the form of tables.

[0050] The data output can be generated automatically or in response to user input. For example, it is possible to generate data output directly when an operation is applied in live mode. The data output can be generated based on the data output of the operator symbol that forms the end of a chain of symbols. If data sets or parameters of operators are changed, these are then re-outputted, whereby, particularly in the case of chains of operators, only the part of the chain affected by the changed parameters is recalculated. However, an offline mode can be provided, in which data symbols and operator symbols can be placed on the user interface without data output being generated. In this case, user input, e.g.By activating a button, data output can be started based on all or specific symbols on the user interface. This offline mode is particularly advantageous when using computationally intensive operators.

[0051] Images and other output data can be displayed, for example, on a data output interface alongside the user interface. This allows the user, especially with many parallel processing threads, to view only selected processing results, which increases clarity and usability. Corresponding output data and processing pipelines (i.e., arrangements of icons on the user interface) can be marked. For example, an image could be graphically highlighted when the user places a mouse cursor over the corresponding processing pipeline, and vice versa.

[0052] According to a further embodiment, the data output comprises a control command for a component of a light microscope or a component connected to the light microscope, such as a flow cell or a pump, for adjusting a sample medium. An example application of this is, for example, controlling the addition of a substance to the sample medium when an operation has detected that biological cells in the sample have certain properties, e.g., are in a specific growth phase.

[0053] According to a further embodiment, the method comprises inserting an output symbol at one of the predefined positions on the user interface, wherein the output symbol represents the data output. The processed light microscopic data set can thus be displayed, for example, as an image on the display unit if the user places an output symbol at the corresponding location on the user interface (in particular adjacent to a data output of an operator symbol).

[0054] Alternatively to the embodiment described above, a data output may be generated, for example, automatically or in response to a user input, e.g. activating a button, only for arrangements of data symbols and operator symbols that additionally contain an output symbol.

[0055] According to a further embodiment, a chain of at least two operator symbols is arranged on the user interface, wherein the data output is automatically generated based on an operation represented by an operator symbol that forms one end of the chain. The term "end" refers to the processing direction, i.e., the corresponding operation is the last operation executed in the processing sequence.

[0056] According to a further embodiment, in response to a user input, a data output is generated based on an operation represented by an operator symbol that is located within the chain (i.e., does not form the end of the chain).

[0057] According to a further embodiment, the method comprises inserting a reference symbol at one of the predefined positions on the user interface, wherein the reference symbol represents an output value of a data symbol or operator symbol. Reference symbols can be inserted elsewhere on the user interface as a data source, particularly through user interaction or automatically. In this way, for example, different processing threads can be linked, which increases flexibility and user-friendliness.

[0058] For example, FOR loops can also be implemented using reference symbols.

[0059] According to a further embodiment, the method comprises inserting a recording symbol at one of the predefined positions on the user interface, wherein the recording symbol represents a light microscopic image, in particular wherein the light microscopic image is taken in response to the insertion of the recording symbol or in response to a user input. Alternatively, the light microscopic image can also be taken when an operation represented by an operator symbol is executed, wherein in particular the operator symbol is arranged on the user interface adjacent to a data input of the recording symbol. According to a further embodiment, a data symbol is automatically inserted at one of the predefined positions of the user interface, wherein the data symbol represents a light microscopic data set obtained by the light microscopic image.Alternatively, the obtained light microscopy data can also be saved, for example, as a subset of a parent light microscopy dataset, represented by a data symbol. The data symbol can be placed, for example, at a previously marked position or at a data output of an acquisition symbol.

[0060] Using the acquisition symbol, automated measurements can be advantageously combined with data processing steps. For example, a region of interest identified in an image during surgery can be automatically analyzed using adjusted parameters or a different light microscopy technique (particularly with higher resolution, such as STED microscopy).

[0061] In particular, the image symbol is assigned measurement parameters that can be adjusted through user input. These parameters can, for example, specify the type of light microscopy image, the intensity of an illumination light, or light detection parameters.

[0062] According to a further embodiment, the method comprises inserting at least two processing threads on the user interface, wherein the processing threads each have at least one data symbol and / or at least one operator symbol.

[0063] According to a further embodiment, each of the processing threads has at least one data symbol and at least one operator symbol, wherein the processing threads represent mutually independent processing sequences of the light microscopic data set represented by the data symbol or the light microscopic data sets represented by the data symbols.

[0064] An example of this application is parallel processing threads, each of which ends with different data objects, but which contain the same operator objects. This allows different data sets to be processed in a clear manner using the same processing steps. Furthermore, parallel processing threads make it possible, for example, to process the same data set in parallel using different operations.

[0065] According to a further embodiment, several of the processing threads are linked by an operator symbol, wherein the operator symbol covers several adjacent predefined positions. In particular, these operator symbols are wider, in particular wider by an integer factor, than the data symbols and / or wider than other operator symbols that only cover one predefined position. The corresponding width runs in a different direction than the processing direction. If the predefined positions are marked visibly on the user interface, "covering" does not necessarily mean that the predefined positions are no longer visible when the operator symbol is located on them. The predefined positions can, for example, still be visible if the operator symbol is displayed as fully or partially transparent.The only important thing is that the operator symbol is located in several predefined positions so that it connects several processing threads.

[0066] According to a further embodiment, the operator symbol covering several of the predefined positions links at least two data symbols together. For example, such an operator symbol can be arranged below or next to two or more data symbols arranged at predefined positions, wherein the predefined positions belong to different processing threads. The operator symbol can, for example, be twice as wide as the data symbols. The data sets represented by the data symbols are then offset, in particular, with the operations represented by the operator symbol. For example, the operations can include calculating the difference between two data sets. The same principle can, of course, also be applied to three or more data sets.

[0067] According to a further embodiment, the operator symbol covering several of the predefined positions links the data outputs of further operator symbols assigned to two adjacent processing threads. In this way, the data outputs of other operator symbols can be offset against each other.

[0068] According to a further embodiment, several of the processing threads are linked by a data symbol, wherein the data symbol covers several predefined positions, in particular in a direction perpendicular to the processing direction. Such a data symbol has, in particular, several data outputs, adjacent to each of which an operator symbol can be arranged.

[0069] According to a further embodiment, the processing threads are formed by an operator symbol that has at least two data outputs. In other words, such an operator forms a branching point from which several processing threads originate. The processing threads can optionally be rejoined by a further operator symbol (e.g., with several data inputs and one data output). A processing pipeline can therefore also be represented on the user interface, in particular, as a circular graph of symbols. According to a further embodiment, several operator symbols are combined into a group, in particular by means of a user input, wherein the group is represented by a group symbol. The group symbol then represents a sequence of operations represented by the several operator symbols.This simplifies the presentation for the user and improves usability through a simple graphical representation of customized, user-defined sequences of operations. In particular, a group symbol can be saved and / or reproduced by the user so that it can be reused elsewhere on the user interface and / or in subsequent applications of the inventive method. A group can be created, for example, by dragging a box on the user interface around several operator symbols. Of course, a group can also comprise other symbols in addition to operator symbols, e.g., output symbols, recording symbols, or reference symbols.

[0070] According to a further embodiment, an operation represented by the at least one operator symbol is recalculated in response to a user input. This results in the computational operations required for processing being performed only when needed. This allows computing capacity to be saved.

[0071] According to a further embodiment, an operation represented by the at least one operator symbol is recalculated if at least one light microscopic data set represented by a data symbol has been modified. In this case, the processing is performed automatically, which increases user-friendliness.

[0072] A second aspect of the invention relates to a system comprising a display unit and a computing unit. The system is designed to display a two-dimensional user interface and to provide predefined positions on the user interface, to insert at least one data symbol representing a light microscopic data set at one of the predefined positions on the user interface, and to insert at least one operator symbol representing an operation performed or to be performed on the light microscopic data set at one or more of the predefined positions on the user interface, wherein the computing unit is designed to apply the operation represented by the operator symbol or the operations represented by the operator symbols to the at least one data symbol orto apply the light microscopic data set represented by the data symbols in a processing sequence determined by an arrangement of the at least one data symbol and the at least one operator symbol on the user interface along a predetermined processing direction. According to a further embodiment, the system is designed to carry out the method according to the first aspect.

[0073] According to a further embodiment, the system comprises an input unit configured to capture user inputs and to cause the computing unit to perform operations based on the user inputs and / or to cause the display unit to display a graphical output based on the user input and / or to cause the display unit to insert data symbols, operator symbols, output symbols and / or reference symbols on the user interface.

[0074] For example, a user can use the input unit to arrange symbols on the user interface and / or adjust parameters of operator symbols.

[0075] A third aspect of the invention relates to a light microscope comprising a system according to the second aspect of the invention.

[0076] According to one embodiment, the light microscope comprises a light source configured to generate an illuminating light beam. The light source may, in particular, be a laser, and the illuminating light may be excitation light that excites emitters in the sample to luminescence, in particular fluorescence.

[0077] According to a further embodiment, the light microscope has an objective lens configured to illuminate a sample with the illumination light beam (in particular, to focus the illumination light beam into the sample). In particular, the objective lens is configured to collect light emanating from the sample, in particular emission light emanating from emitters in the sample (furthermore, in particular, luminescence light, furthermore, in particular, fluorescence light).

[0078] According to a further embodiment, the light microscope has a detector which is designed to detect light emanating from the sample, in particular emission light emanating from emitters in the sample.

[0079] According to a further embodiment, the light microscope is designed to generate at least one light microscopic data set, wherein the computing unit is designed to process the light microscopic data set using the method according to the first aspect. The light microscopic data set is generated, in particular, by converting light intensities detected by the detector or photon numbers of light emanating from the sample into data form by a computing unit and storing the data set on a volatile or non-volatile storage medium. The light microscope can be, for example, a laser scanning microscope, a wide-field fluorescence microscope, a STED microscope, or a localization microscope (in particular a MINFLUX microscope).

[0080] A fourth aspect of the invention relates to a computer program comprising instructions that cause the system according to the second aspect or the light microscope according to the third aspect to execute the method according to the first aspect. The computer program can be implemented in hardware or software and stored on a volatile or non-volatile storage medium. It can be executed, for example, on the computing unit of the system according to the invention.

[0081] Further embodiments of the system according to the second aspect, the light microscope according to the third aspect, and the computer program according to the fourth aspect emerge from the embodiments of the method according to the first aspect presented above. Conversely, further aspects of the method according to the first aspect and the computer program according to the fourth aspect emerge from embodiments of the system according to the second aspect and the light microscope according to the third aspect.

[0082] Advantageous developments of the invention will become apparent from the patent claims, the description, the drawings, and the accompanying explanations of the drawings. The described advantages of features and / or combinations of features of the invention are merely exemplary and may be effective alternatively or cumulatively.

[0083] With regard to the disclosure content (but not the scope of protection) of the original application documents and the patent, the following applies: Further features can be found in the drawings – in particular the relative arrangements and operative connections shown. The combination of features of different embodiments of the invention or of features of different patent claims is also possible, deviating from the selected references of the patent claims, and is hereby suggested. This also applies to features that are shown in separate drawings or mentioned in their description. These features can also be combined with features of different patent claims. Likewise, features listed in the patent claims can be omitted for further embodiments of the invention; however, this does not apply to the independent patent claims of the granted patent.

[0084] The reference symbols contained in the patent claims do not limit the scope of the subject matter protected by the patent claims. They serve merely to make the patent claims easier to understand. Exemplary embodiments of the invention are described below with reference to the figures.

[0085] These do not limit the subject matter of this disclosure and the scope of protection.

[0086] Short description of the characters

[0087] Fig. 1 shows an embodiment of a light microscope according to the invention with a system according to the invention comprising a display unit and a computing unit for carrying out a method according to the invention;

[0088] Fig. 2 shows a first embodiment of a user interface displayed by means of the display unit of the system according to the invention;

[0089] Fig. 3 shows a second embodiment of a user interface displayed by means of the display unit of the system according to the invention;

[0090] Fig. 4 shows a third embodiment of a user interface displayed by means of the display unit of the system according to the invention;

[0091] Fig. 5 shows a fourth embodiment of a user interface displayed by means of the display unit of the system according to the invention;

[0092] Fig. 6 shows a fifth embodiment of a user interface displayed by means of the display unit of the system according to the invention;

[0093] Fig. 7 shows a sixth embodiment of a user interface displayed by means of the display unit of the system according to the invention;

[0094] Fig. 8 shows a seventh embodiment of a user interface displayed by means of the display unit of the system according to the invention.

[0095] Description of the characters

[0096] Fig. 1 shows a light microscope 1 with a system according to the invention comprising a display unit 8 and a computing unit 9. The light microscope 1 has a light source 2 (in particular a laser) for generating an illuminating light beam B. The illustrated embodiment is a confocal laser scanning microscope. Of course, the invention is not limited thereto. The illuminating light beam B passes through a beam splitter 6 (e.g., a dichroic mirror) and a scanning device 5 (e.g., a galvanometer scanner) and is then focused by an objective 3 into a sample P. The scanning device 5 shifts the focus of the illuminating light beam B in the sample P, in particular laterally in two mutually perpendicular directions, i.e., perpendicular to an optical axis of the objective 3.

[0097] The sample P contains emitters that emit light when irradiated with the illumination light. The emitted light is, in particular, luminescent light, and more particularly, fluorescent light, with the illumination light exciting the emitters in the sample P. However, the emitters can also be, for example, light-scattering particles. In this case, the emitted light is scattered light.

[0098] The emission light emanating from the sample P is collected by the objective lens 3, descanned by the scanning device 5, and reflected by the beam splitter 6 into a detection beam path. The reflected emission light is detected by a detector 4 (e.g., a point detector such as an avalanche photodiode or a photomultiplier) and registered by a computing unit 9 (part of the system according to the invention) coupled to the detector 4. A light microscopic data set 20 is created from the signal of the detector 4.

[0099] A display unit 8, e.g., a screen, is coupled to the computing unit 9, on which a user interface 10 is displayed according to the inventive method. A user can place data symbols 13 and operator symbols 14 on this user interface 10 by means of an input unit 7 coupled to the computing unit 9, wherein the data symbols 13 represent light microscopic data sets 20, and wherein the operator symbols 14 represent processing operations.

[0100] The light microscope 1 shown in Fig. 1 can, for example, also be modified so that the emission light is not descanned. To do this, the beam splitter 6 must be placed between the objective 3 and the scanning device 5 (so-called non-descanned configuration).

[0101] An alternative embodiment (not shown) of the light microscope 1 according to the invention is a wide-field microscope. In this case, the illumination light beam B is not focused by the objective 3 into the sample P, but rather illuminates a section of the sample P approximately homogeneously. The detector 4 in this case is, in particular, a spatially resolving detector with a plurality of pixels, e.g., a CCD or CMOS camera.

[0102] The light microscope 1 can also be a superresolution microscope (not shown). For this purpose, in addition to the light source 2, an additional STED laser and a light modulator (e.g., a phase plate or a so-called spatial light modulator, SLM) can be provided to form a STED light distribution with a central minimum at the focus.

[0103] A further alternative embodiment is a localization microscope, e.g., a MINFLUX microscope (not shown). For this purpose, a light modulator is also provided, which shapes the illumination light (in this case, excitation light) at the focus into a light distribution with a central minimum. Optionally, a further fast beam shifting unit with electro-optical deflectors is provided. In this case, a control unit is also provided, which shifts the light distribution at the focus to various scanning positions near a presumed position of an individual emitter. The computing unit is designed, in particular, to calculate a position of the emitter from the emission light detected for various scanning positions.

[0104] Fig. 2 shows an embodiment of a user interface 10 displayed on a display unit 8, with which the method according to the invention can be carried out. Predefined positions 11 are arranged on the user interface 10 as a two-dimensional grid 15. In the example shown, the predefined positions 11 are marked by crosses 11a, at which corners of rectangular data objects 13 and operator objects 14 can be placed. The respective predefined position 11 is then a center point between four such crosses 11a.

[0105] Furthermore, Fig. 2 shows a processing direction V which runs from top to bottom.

[0106] At the upper ends of two parallel processing lines 12 are two data symbols 13, each representing a light microscopic data set 20. These can, for example, be images of the same area of ​​sample P in different color channels. In both processing lines 12, an operator symbol 14 is shown below the respective data symbol 13. Here, this symbol represents, for example, a baseline correction, in which a value, which may be, for example, background emission, is subtracted from the intensities of all pixels. This value is symbolized in the operator symbols 14 by the dashed line.Below the operator symbols 14 of the two processing threads 12 there is each an output symbol 15, the placement of which on the user interface 10 in the embodiment shown here results in the display of an image by the output unit 8, wherein the image is the result of the processing of the respective light microscopic data 20 with the processing operation represented by the operator symbol 14 in the respective processing thread 12. In the example shown in Fig. 2 with parallel processing threads 12, for example, two images can be displayed next to one another or two superimposed images. The images can be displayed, for example, in an image display interface (not shown) arranged next to the user interface 10. As an alternative to the output based on the output symbols 16, a processed image can be displayed automatically, for example, directly after the placement of a new operator symbol 14 in a processing thread 12.

[0107] Fig. 3 shows a further embodiment of a display unit 8

[0108] User interface 10 with two processing threads 12 coupled via an operator symbol 14. The predefined positions 11 are arranged in a Cartesian grid 15 on the user interface 10, as in the embodiment shown in Fig. 2. The operator symbol 14 represents a difference formation between two light microscopic data sets 20, each represented by data symbols 13, and has two data inputs 18 and one data output 19. The operator symbol 14 covers two of the predefined positions 11 in a direction perpendicular to the processing direction V and in this way links two processing threads 12. The data symbols 13 are arranged in the two adjacent processing threads 12 above the operator symbol 14 adjacent to the data inputs 18.Below the data output 19, as part of the left processing thread 12, an output symbol 16 is arranged, the placement of which on the user interface 10 leads to the display of an image which is a result of the difference formation between the two light microscopic data sets 20 represented by the data symbols 13.

[0109] A further embodiment of a user interface 10 displayed by a display unit 8 is shown in Fig. 4. The predefined positions 11 are arranged in a Cartesian grid 15 on the user interface 10, as in the embodiment shown in Fig. 2. In the embodiment of the method according to Fig. 4, the operations of several operator symbols 14 are combined in three processing threads 12. At the upper end of the arrangement of data symbols 13 and operator symbols 14, similar to the embodiment according to Fig. 3, a first data symbol 13a and a second data symbol 13b (which can represent, for example, two color channels of an image of the same sample section) are placed at the data inputs 18 of a first operator symbol 14a (masking operator symbol), which covers two predefined positions 11 arranged next to one another and thus two processing threads 12.The first operator symbol 14a here represents an operation in which a mask is generated from one of the light microscopic data sets 20, which mask is then applied to the other light microscopic data set 20. In this way, for example, regions of interest can be selected in one color channel, wherein the regions are selected based on a different color channel. A second operator symbol 14b (identity operator symbol) is placed at the data output 19 (i.e., in one of the processing threads 12 below the data output 19) of the first operator symbol 14. The second operator symbol 14b has a data input 18 and a data output 19. The operation underlying the second operator symbol 14b duplicates the data output at the data input 18 (i.e., in the processing thread 12 above the data input 18) and outputs it at the data output 19. Next to the second operator symbol 14b, a third data symbol 13c is placed, which, for example,can represent a further color channel, and below it is a third operator symbol 14c (addition operator symbol), wherein the third data symbol 13c and the second operator symbol 14b are arranged at the data inputs 18 of the third operator symbol 14c. The operation underlying the third operator symbol 14c overlays the light microscopic data set 20 represented by the third data symbol 13c with the processed data set generated by means of the first operator symbol 14a. Below the data output 19 of the third operator symbol 14c is an output symbol 16, the placement of which on the user interface 10 results in the display of the corresponding overlaid image.

[0110] Fig. 5 shows a user interface 10 with a data symbol 13 and an operator symbol 14 (addition operator symbol) that has a handle 21 with which the operator symbol 14 can be widened so that it covers additional predefined positions 11. This can be done, for example, by a drag-and-drop operation via a mouse click or by swiping on a touch-sensitive screen. In this way, for example, three or more light microscopic data sets 20 can be overlaid with the addition operator in one image.

[0111] Fig. 6 illustrates a further embodiment of a user interface 10 with reference symbols 17. In a processing thread 12, a data symbol 13 and a first operator symbol 14a are arranged one below the other. Another processing thread 12 consists of a reference symbol 17, a second operator symbol 14b, and an output symbol 16, which are arranged one below the other. The reference symbol 17 transfers the processed data set output at the data output 19 of the first operator symbol 14a to the data input 18 of the second operator symbol 14b. After the operation represented by the second operator symbol 14b has been performed, a processed image is displayed. In this way, different processing threads 12, even those that are further apart and not adjacent on the user interface 10, can be linked to one another.

[0112] Fig. 7 shows a further embodiment of a user interface 10 with a processing thread 12 which comprises a first data symbol 13a, an operator symbol 14, a recording symbol 22, a second data symbol 13b and a display symbol 16.

[0113] In particular, based on an operation represented by operator symbol 14, parameters of a light microscopic image represented by acquisition symbol 22 are automatically determined. For example, the operation may be an automatic selection of a region of interest from a lower-resolution overview image. In response to a user input in which acquisition symbol 22 is placed below operator symbol 14 in the processing direction V (in particular at a data output 19 of operator symbol 14), an light microscopic image of the region of interest is automatically started using a technique that provides better resolution than the overview image. The overview image can be acquired, for example, using confocal laser scanning microscopy and the higher-resolution image using STED microscopy.

[0114] Subsequently, the second data symbol 13b is automatically generated and placed under the recording symbol 22, wherein the second data symbol 13b represents the light microscopic data 20 obtained during the recording started by the recording symbol 22.

[0115] Arranging the output symbol 16 below the second data symbol 13b then results in the display of the light microscopic data 20 represented by the second data symbol 13b as an image.

[0116] Fig. 8 shows an embodiment in which a first operator symbol 14a creates a branch into two processing threads 12. The first operator symbol 14a has a data input 18 and two data outputs 19. A data symbol 13 representing a light microscopic data set 20 is arranged above the data input 18. A second operator symbol 14b and a third operator symbol 14c, which perform baseline corrections in this case, are arranged below the data outputs 19 in the processing direction V.

[0117] The first operator symbol 14a can represent a pure branching operation in which the light microscopic data set is duplicated and passed to both data outputs 19. In addition, however, two different operators can also be applied to the light microscopic data set 20 and output at the data outputs 19.

[0118] List of reference symbols

[0119] 1 light microscope

[0120] 2 light source

[0121] 3 Lens

[0122] 4 Detector

[0123] 5 Scanning device

[0124] 6 beam splitters

[0125] 7 Input unit

[0126] 8 Display unit

[0127] 9 Computing unit

[0128] 10 User interface

[0129] 11 Predefined positions

[0130] 11a Cross

[0131] 12 processing lines

[0132] 13 Data symbol

[0133] 13a First data symbol

[0134] 13b Second data symbol

[0135] 13c Third data symbol

[0136] 14 Operator symbol

[0137] 14a First operator symbol

[0138] 14b Second operator symbol

[0139] 14c Third operator symbol

[0140] 15 grids

[0141] 16 Output symbol

[0142] 17 Reference symbol

[0143] 18 Data input

[0144] 19 Data output

[0145] 20 Light microscopic data set

[0146] 21 handles

[0147] 22 Recording symbol

[0148] B Illumination light beam

[0149] P Sample

[0150] V Processing direction

Claims

Patent claims 1. Method for processing light microscopic data comprising the steps: - Providing predefined positions (11) on a two-dimensional user interface (10) displayed by means of a display unit (8); - inserting at least one data symbol (13) representing a light microscopic data set (20) at one of the predefined positions (11) on the user interface (10); - inserting at least one operator symbol (14) representing an operation to be performed on the light microscopic data set (20) at one of the predefined positions (11) on the user interface (10); - applying the operation or operations represented by the at least one operator symbol (14) to the at least one light microscopic data set (20) represented by the data symbol (13) or the data symbols (13) in a processing sequence that is determined by an arrangement of the at least one data symbol (13) and the at least one operator symbol (14) on the user interface (10) along a predetermined processing direction (V).

2. Method according to claim 1, characterized in that the at least one data symbol (13) has at least one data output (19) and / or that the at least one operator symbol (14) has at least one data input (18) and at least one data output (19).

3. Method according to claim 2, characterized in that the at least one data input (18) and / or the at least one data output (19) is graphically identified.

4. Method according to one of the preceding claims, characterized in that the predefined positions (11) form a regular two-dimensional grid (15).

5. Method according to one of the preceding claims, characterized in that the processing direction (V) is vertical or horizontal.

6. Method according to one of the preceding claims, characterized in that the operations are selected from the group: arithmetic basic operations, intensity scaling, linear image deconvolution, iterative image deconvolution, noise reduction / removal, thresholding, masking, image segmentation, demixing, automatic object detection, resampling, color adjustment, brightness adjustment, adjustment of a display area, coordinate transformation.

7. Method according to one of the preceding claims, characterized in that the method comprises generating a data output based on the applied operation or operations, in particular wherein the data output comprises displaying at least one processed image, diagram, geometric shape, text or at least one number or saving or exporting a processed data set.

8. The method according to claim 7, characterized in that the method comprises inserting an output symbol (16) at one of the predefined positions (11) on the user interface (10), wherein the output symbol (16) represents the data output.

9. Method according to claim 7 or 8, characterized in that a chain of at least two operator symbols (14) is arranged on the user interface (10), wherein the data output is automatically generated on the basis of an operation represented by an operator symbol (14) forming one end of the chain.

10. The method according to claim 9, characterized in that in response to a user input, a data output is generated based on an operation represented by an operator symbol (14) arranged within the chain.

11. Method according to one of the preceding claims, characterized in that the method comprises inserting a reference symbol (17) at one of the predefined positions (11) on the user interface (10), wherein the reference symbol (17) represents an output value of a data symbol (13) or operator symbol (14).

12. Method according to one of the preceding claims, characterized in that the method comprises inserting a recording symbol (22) at one of the predefined positions (11) on the user interface (10), wherein the recording symbol (22) represents a light microscopic recording, in particular wherein the light microscopic recording is carried out in response to the insertion of the recording symbol (22) or in response to a user input, further in particular wherein a data symbol (13) is automatically inserted at one of the predefined positions (11) of the user interface, wherein the data symbol (13) represents a light microscopic data set (20) obtained by the light microscopic recording.

13. Method according to one of the preceding claims, characterized in that the method comprises inserting at least two processing threads (12) on the user interface (10), wherein the processing threads (12) each comprise at least one data symbol (13) and / or at least one operator symbol (14).

14. The method according to claim 13, characterized in that each of the processing strands (12) has at least one data symbol (13) and at least one operator symbol (13), wherein the processing strands (12) represent mutually independent processing sequences of the light microscopic data set (20) represented by the data symbol (13) or the light microscopic data sets (20) represented by the data symbols (13).

15. The method according to claim 13, characterized in that several of the processing strands (12) are linked by an operator symbol (14), wherein the operator symbol (14) covers several predefined positions (11), in particular in a direction perpendicular to the processing direction (V).

16. The method according to claim 15, characterized in that the operator symbol (14) covering a plurality of predefined positions (11) links at least two data symbols (13) to one another.

17. Method according to claim 15, as far as dependent on claim 2, characterized in that the operator symbol (14) covering several of the predefined positions (11) links the data outputs (19) of further operator symbols (14) which are assigned to two adjacent processing strands (12).

18. The method according to claim 13, characterized in that several of the processing strands (12) are linked by a data symbol (13), wherein the data symbol (13) covers several predefined positions (11), in particular in a direction perpendicular to the processing direction (V).

19. Method according to claim 13, as far as dependent on claim 2, characterized in that the processing strands (12) are formed by an operator symbol (14) which has at least two data outputs (19).

20. Method according to one of the preceding claims, characterized in that several operator symbols are combined to form a group, the group being represented by a group symbol.

21. Method according to one of the preceding claims, characterized in that an operation represented by the at least one operator symbol (14) is recalculated in response to a user input.

22. Method according to one of the preceding claims, characterized in that an operation represented by the at least one operator symbol (13) is recalculated if at least one light microscopic data set (20) represented by a data symbol (13) has been changed.

23. System comprising a display unit (8) and a computing unit (9), wherein the system is designed to display a two-dimensional user interface (10) on the display unit (8) and to provide predefined positions (11) on the user interface (10), to insert at least one data symbol (13) representing a light microscopic data set (20) at one of the predefined positions (11) on the user interface (10) and to insert at least one operator symbol (14) representing an operation performed or to be performed on the light microscopic data set (20) at one of the predefined positions (11) on the user interface (10), wherein the computing unit (9) is designed toto apply the operation represented by the operator symbol (14) or the operations represented by the operator symbols (14) to the at least one light microscopic data set (20) represented by the data symbol (13) or the data symbols (13) in a processing sequence that is determined by an arrangement of the at least one data symbol (13) and the at least one operator symbol (14) on the user interface (10) along a predetermined processing direction (V).

24. System according to claim 23, characterized in that the system comprises an input unit configured to capture user inputs.

25. Light microscope (1) comprising a system according to claim 23 or 24.

26. A computer program comprising instructions which cause the system according to claim 23 or 24 or the light microscope (1) according to claim 21 to carry out the method according to one of claims 1 to 22.

Images