How to record the workflow of a scientific experiment

Abstract

The method for recording and analyzing a scientific experiment workflow comprises making a video recording comprising a number of successive individual images of at least a part of a workspace in which the workflow is being executed during the course of the workflow, and at the same time storing the values ​​of parameters relevant to the workflow as separate data sets (31, 32, 33, 34, 35) in a digital data bank (30). The data sets (31, 32, 33, 34, 35) comprise the individual images of the video recording and the values ​​of the parameters over time, which are assigned to a common reference time (38), so that at each time point there is a clear temporal relationship between the individual images of the video recording and the values ​​of the parameters. At least the parameter data sets are stored in a searchable form to allow searching for parameter events indicating the retrieved values ​​or retrieved changes of the parameters. The values ​​of at least selected parameters and the images of the video recording are preferably visually displayed in temporal relation to each other.

Description

[Technical field]

[0001] The recording of chemical or other scientific experiments and workflows is typically performed in the form of a user-manually recorded recording of tasks, measurements and observations. Thus, a user (e.g. a laboratory worker) records, for example, the individual work steps of a workflow as a sequence of work steps and tasks performed in chronological order, as well as measurements and observations. [Background technology]

[0002] In automated operating devices, which perform scientific experiments and / or manufacturing processes individually, sequentially or in parallel, a large number of parameters are also usually recorded. For example, sensor measurements or tasks performed by the device are automatically recorded, again with recording being performed sequentially (also parallel in the case of parallel workflows). If the corresponding operating device is equipped with several sensors, measuring devices or otherwise measurable parameters, the respective individual measurements / recordings are likewise recorded consecutively. Here, the term "parameter" means in each case values ​​determined in an experiment or by a method or device, planned values, planned or performed actions, any form of recording, but also values ​​observed, for example, by a laboratory worker or the subjective observation / interpretation of an observer.

[0003] In such cases, the records thus created are therefore a series of individual measurements, records and observations, and the evaluation of that data and information as a whole often requires time-consuming and laborious manual or visual filtering and searching for the required information. In particular, the (temporal) linking of that data and information to the workflow records, and the clear indication of such linking, remains unresolved in the prior art.

[0004] In the laboratory, every scientific experiment or workflow generates a huge amount of data in a huge variety of formats and types, and the amount of data grows as new workflows are constantly performed. Searching through that data for individual required parameters, values ​​or pieces of information has traditionally been a big challenge for laboratory workers, and even in the case of information available in the form of video recordings, it has been virtually impossible to link that data to visual inputs in the recording. For example, in the case of a video recording of a workflow, it is necessary to review the entire recording in order to identify and interpret the event of interest (e.g., color change, overheating, bubbling, etc.) and link that event to another data series (e.g., temperature series) in order to interpret the relationship.

[0005] The visual recording of experiments using cameras has the major drawback that even with the best AI systems, the search for specific data can only be performed in a very limited way and it is difficult to connect in time a specific image sequence to data recorded using other recording techniques, sensors, etc.

[0006] The aim of the invention is to simplify and optimize the recording and logging of all relevant data involved in workflows, especially those involving video recording, IR, UV or other recording techniques, or other complex recording methods (audio, ultrasound, radioactivity), allowing easy retrieval of parameters and events throughout the entire workflow, some of which may be very long or may span several different workflows.

[0007] In particular, the object of the invention is to simplify the overall recording and logging of all relevant measurable and observable parameters of the operations and devices involved in the workflow of a scientific experiment, as well as of user actions and observations, so that a subsequent analysis of the entire information can be performed in as simple, fast and targeted a manner as possible, and in such a way that all these data can be directly connected at any time to a video recording of the workflow.

[0008] The problem underlying the present invention is solved by the method according to the invention as defined in the independent claims 1 and 2. Advantageous configurations and developments are the subject of the dependent claims. Summary of the Invention

[0009] The essence of the method according to the invention for recording the workflow of a scientific experiment is the following. In a method for recording a scientific experiment workflow, a video recording is made during the course of the workflow, comprising a number of successive individual images of at least a portion of a workspace in which the workflow is performed, and at the same time, values ​​of parameters relevant to the workflow are recorded and stored as individual data sets in a digital data bank. The data sets comprise the individual images of the video recording and the values ​​of the parameters over time, assigned to a common reference time, so that at each time point there is a clear temporal association between the individual images of the video recording and the values ​​of the parameters. The parameter data sets are stored in a searchable form, so that it is possible to search for parameter events in which the parameters indicate a searched value or a searched change.

[0010] The essence of the method according to the invention for recording and analysing the workflow of a scientific experiment is the following. In a method for recording and analyzing a scientific experiment workflow, a video recording is made comprising a number of successive individual images of at least a part of a workspace in which the workflow is being executed during the course of the workflow, and at the same time, values ​​of parameters relevant to the workflow are recorded and stored as individual data sets in a digital data bank. The data sets comprise the individual images of the video recording and the values ​​of the parameters over time, which are assigned to a common reference time, so that at each time point there is a clear temporal association between the individual images of the video recording and the values ​​of the parameters. The parameter data sets are stored in a searchable form, so that it is possible to search for parameter events in which the parameters indicate a search value or a search change. Values ​​of at least selected parameters present at the time of the search parameter event or present in a period including the time of the search parameter event and the images of the video recording are visually displayed in a time-related relationship with each other.

[0011] The method according to the invention allows a simple, fast and targeted search of recorded video with reference to synchronously recorded parameter values ​​and vice versa.

[0012] In the method according to the invention, the recording or storage of all the values ​​of the parameters in separate data sets is performed synchronously in the same way as a movie soundtrack, only in the case of a scientific experiment workflow, where often several hundred such data sets are recorded. In each of these data sets, a search is performed according to certain criteria and immediately linked to the video recordings, i.e. the corresponding video sequence is automatically and finally assigned. In this way, the video recordings of the scientific experiment workflow (or the recordings in different wavelength ranges) can be efficiently searched and it is possible to visually display or track what happened at the moment when a certain parameter value or a change in a certain value occurred in the workflow and / or what exactly was being operated at or before that moment.

[0013] What is important for the invention here is that for each individual measurable or observable parameter a dedicated data set is created in which all values ​​or value changes of the parameter are recorded and / or their derivatives (e.g. temperature gradient as a derivative of temperature). Thus, a data set of a temperature sensor records, for example, the respective measured temperature or temperature change as well as the time of the temperature measurement or temperature change, or a data set of an automatic metering device records at what time what action was taken (metering of a desired substance into a desired object) and what result was obtained (here, for example the amount actually metered). The combination of the data sets shows how the workflow was performed and, together with the visual display, a searchable visual record is provided.

[0014] It will be appreciated that time-accurately correlated data series can also be transformed, for example, into other values ​​or statistically smoothed. Outliers (e.g., measurements occurring for less than 0.1 seconds) can be removed. Or new time-adjusted data series can be created that calculate the difference between two data sets or, for example, the increase in speed. These transformed data series can also be searched to assign and display the corresponding video recordings.

[0015] In the method according to the invention, these individual parameters or data sets are displayed in a manner similar to that known for example in the music or movie industry, each parameter being displayed as a "track". All data sets are recorded synchronously, i.e. temporally associated, and selected or all data sets are displayed visually, for example on a screen.

[0016] According to the invention, it is important that all recorded data sets are time-synchronized, which is essential in order to be able to later find and, above all, visually track the time points of individual measurements, observations, recordings and actions in relation to the totality of all measurement data. This allows, first of all, "visual recording", since properties such as, for example, parameter values ​​or parameter value changes can be easily searched for. Time referencing can be performed relative to a desired point in time (e.g. the start of a workflow), with each measurement / recording associated to that point in time.

[0017] Advantageously, the recording times refer to a universal timescale, e.g. Universal Time (UTC) or International Atomic Time (TAI), and each time point of the measurement refers to a corresponding universal time axis, so that each individual data point can then be unambiguously and precisely related in time.

[0018] Advantageously, for selected parameters, the target values ​​of those parameters are stored in a data bank as a data set and visually displayed together with the recorded actual values ​​of those parameters. Advantageously, the change in value over time of at least the selected parameters, if applicable together with the corresponding target values, is visually displayed in adjacent or one above the other arranged tracks, and the change in the value of at least the selected parameters over time, preferably together with the associated target values, is visually displayed in the form of a numerical or graphical line. This kind of display facilitates an overview of the recorded parameter values.

[0019] Advantageously, the individual data sets are graphically displayed as parallel tracks integrated into a visual record of the workflow or a parallel visual record (video) in such a way as to show a table running synchronously along a common time axis with the visual display (video) above the video or next to the video record.

[0020] It will be appreciated that the display of the tracks can also be arranged side-by-side horizontally (the tracks themselves run vertically and the time axis runs vertically, e.g., from top to bottom) or arranged vertically one after the other (the tracks themselves run horizontally and the time axis runs horizontally, e.g., from left to right). Display across multiple screens or different output devices (e.g., a combination of screens and an augmented reality display device (Microsoft's HoloLens)) is also possible.

[0021] Advantageously, two or more video recordings of the workspace are made at different viewing angles and / or different wavelength ranges and stored as separate data sets. Advantageously, radiation values, e.g., radioactivity or x-ray emissions, magnetic field values, airflow values, or ultrasound measurements, are also stored as separate data sets and may be displayed one above the other for more efficient or intuitive playback, retrieval, or display.

[0022] Video recordings from different viewing angles can show details of the workflow, can be displayed as an overlay or partial overlay, can be combined, for example, to form a three-dimensional display, and can also be additionally displayed, for example, using an augmented reality device.

[0023] Video recordings in different wavelength ranges (e.g. visible, UV or IR) as well as other radiation values, magnetic field values, airflow values ​​or ultrasound measurements provide additional information about the workflow of a scientific experiment.

[0024] Advantageously, the values ​​of at least selected parameters are entered into the databank together with each time point, either automatically or manually by the user.

[0025] The user can also mark (tag) desired images or video sequences as well as values ​​or events of data tracks to make them easier to find later across recordings and parameters. Such marking can also be performed by automation in some cases. In automation, the desired parameters are identified from complex data tracks such as video or audio recordings by image or voice recognition methods. Time points or sequences are marked accordingly (e.g., specific colors, color changes, sounds, other wavelengths, or values, patterns, and characteristics detectable by automation). In an IR video recording, it is possible to mark the sequence in which a specific maximum temperature was detected. Of course, it is also possible to mark multiple data sets or tracks, which in particular offers the ability to create links between data tracks and the events recorded therein.

[0026] Advantageously, the observations recorded by the user in text form, or preferably the parameterized acoustic recordings, are stored as separate data sets, so that the user's verbalized thoughts, observations and interpretations of the events observed can also be parameterized and assigned to data tracks.

[0027] Thus, parameters other than those obtained from technical devices can also be recorded and stored. These can be, for example, manual tasks and / or observations of a laboratory worker working with manual, semi-automatic or automatic equipment. For example, at some point in an automated workflow, a manual addition of a substance can be performed that cannot be performed with an automated laboratory device. The experiment can be recorded (photographed) in the above-mentioned synchronized manner, for example at three different wavelengths (for example, UV, visible, IR) using glasses like Hololens. Or, for example, a user records observations on the device made while the workflow is proceeding automatically, together with the time of the observation. Observations and, for example, audio inputs made by the user are likewise recorded in one or several separate datasets. Whether related to automatic recordings (for example, with a sensor) or manual inputs, it is always important that everything is synchronized with the video recording of the experiment at a particular wavelength (or several video recordings from different viewing angles or wavelengths) and that each type of such recording is recorded separately in a dataset (which, of course, technically can also be individually searchable overlay datasets). The planning data (of the scientific experiment workflow), i.e. both the planning protocol and the (target) data expected at certain points in time, etc., are also stored in a searchable format as a dedicated dataset in exactly the same format, synchronized with the video recordings, so that the video sequences and values ​​of other parameters belonging to the searched (and found) planning data (e.g. target temperature) can be displayed.

[0028] Aggregated data can be performed synchronously by grouping and combining appropriate data sets, scaling or transforming them to further simplify the display, reproduce more intuitive information, or reproduce summary information (e.g. pressure is calculated from temperature and other more or less constant data). For example, a temperature control device such as a cryostat may contain many individual information items, each of which may have its own data set, such as the temperature control target value at a given time, the actual measured temperature, but also the flow rate of the temperature control medium, system information (warning / error messages), control parameters, or simply whether a particular element of the device (heating element or pump of the temperature control medium) is on or off. The recording of all these individual parameters can be stored in a data set as well (e.g. T0: start of pump, T0+1s: measurement of temperature 25.3°C, T0+2s: start of heating element, T0+9s: measurement of temperature 25.5°C, ...) or the individual sub-parameters (here, for example, a) pump activity, b) heating element activity and c) temperature sensor measurement) are recorded as separate data sets, but the data sets can also be later combined and displayed, either combined in one track or "opened" in several tracks. It will be understood that the user can select in the display only those data sets that are useful for the desired analysis. Thus, the user can flexibly configure the display according to the current needs, for example, a video image is displayed on one side of the screen and the desired data tracks of the desired parameters are displayed on the other side, the focus of these data tracks being in each case on the time point displayed in the video image. The visualization of the additionally displayed data tracks advantageously also shows the desired areas before and after the time point currently displayed in the video image, allowing the user to show at a glance the parameter values ​​both before and after the current time point. Thus, users are provided with valuable insight into what happened before the event (and what may have led to the event) and what happened after the event (and what was caused or at least influenced by the event).

[0029] Furthermore, certain parameters are advantageously reproduced in visual form, whether in the form of a warning signal when a certain parameter value is exceeded, or simply displaying the temperature in a color gradient instead of a numerical value (e.g., changing from blue to red depending on the temperature). It will be appreciated that data can come not only from sensors and visual or acoustic recordings, but also that the experimenter can input visual observations (e.g., touch, bubbling, or other parameters that cannot be captured by the available sensors or cameras). These inputs are also captured and synchronously stored (e.g., in the form of audio recordings or written inputs by adding symbols such as marks, exclamation points, smileys, asterisks, etc.) along with their exact time.

[0030] Advantageously, planning data of a scientific experiment workflow are acquired in a similar way in synchrony with the visual recording and stored in a dedicated dataset. These planning data are preferably displayed in local association with the corresponding display of the actual data recorded during the workflow. In this way, search commands can be entered; for example, "Show me the visual recording where the actual value of a recorded parameter (e.g. temperature) for more than 5 seconds, 20 seconds before it reaches a point matching the planned value, and 30 seconds after it reaches that point" or "Show me the video recording that starts 10 seconds before the planned addition of substance X and ends 20 seconds after the effective addition of substance X (and vice versa, if possible)."

[0031] It is not possible to search the data set of the data bank only for parameter values ​​or value changes (not resulting from the video recording). Advantageously, known image recognition methods are used to search the video recording for specific image content or changes in image content. The occurrence or change of the searched image content or a selectable period before and after the occurrence of the searched image content or its change, and the values ​​of at least selected parameters that are mutually associated in time with the images of the video recording are then visually displayed in time. In this way, it is also possible to search backwards. For example, "search for the point where the color (of the reactor contents) changed from yellow to blue, display the changes in the parameters of pressure and temperature 2 minutes before and 5 minutes after the change, while running the video recording in parallel".

[0032] An important advantage of the method according to the invention is that the entirety of the recorded data and information is easily searchable, analysable and displayable in context, for example by selecting a desired time point or a desired period and then selecting all the information, and the recording tracks temporally associated with that time point or period are displayed (filtered, of course, if necessary) or a search for a particular parameter is performed (for example all additions, all additions of a particular substance, all additions to a particular target vessel, all error messages, all temperature measurements showing a temperature higher than a particular value, etc.). Advantageously, this can be used not only in the case of individual scientific experiment workflows, but also across several comparable scientific experiment workflows that are run in parallel or successively, which workflows can be displayed on one or more screens or output devices. What is important here is mainly that in this way the visual record of the workflow can be searched with reference to the measurement data and that the corresponding visual record of the workflow can be used as an efficient record and efficiently searched, even for the most complex workflows. The workflows are equipped with certain features and conclusions can be drawn regarding the visual record at any time. Especially since many workflows can last for hours or days, a visual record of the workflow alone, without an additional data set of further parameters, is of very limited value.

[0033] In an advantageous configuration of the invention, the totality of data and information previously recorded during the workflow is evaluated and interpreted by software designed for that purpose in order to combine parts of the totality of information relevant for further applications and make them further exploitable. In a further advantageous configuration, this can even include a partially or fully automated creation of work instructions for future processing operations based on the already executed workflow.

[0034] In a further advantageous configuration of the invention, for two or more comparable scientific experimental workflows, at least one video recording is made of at least a part of the workspace in which each workflow is executed over the course of each workflow, while values ​​of parameters relevant to each workflow are recorded and stored in a searchable format as separate data sets in a digital data bank and are preferably visually displayed in temporal relation to each other.

[0035] For example, different scientific experiment workflows (e.g., experiments running in parallel) can be displayed in sync with each other (e.g., "display all experiments and associated time points where yellow was maintained for at least 10 seconds, and run the corresponding video recordings before and after that time point").

[0036] As a common reference time between the recordings of several scientific experimental workflows, not only a common relative time point (e.g. the start time in case of experiments started at different times) but also the arrival or occurrence of a desired event, e.g. the time of execution of a particular event or the time of execution of a particular action, can be advantageously used. For example, if several similar workflows are executed one after the other, the common time point can be adjusted to the respective time point of the recording, e.g. when the addition to the vessel is completed, when the temperature is reached, etc. This is also advantageous, for example, if an experiment has several reaction vessels, which have to be filled one after the other due to technical constraints, but the reactions in the vessels need to be compared with each other. In that case, the data tracks of the various sub-experiments are "shifted" and executed in parallel in relation to the reaction-related actions (here, e.g. the addition of a substance to start the reaction) during the execution. Of course, it would be advantageous to be able to switch between different ways of time representation, parallelization and normalization to a common desired relative or absolute time scale.

[0037] Advantageously, the scientific experiment workflow so recorded can also be played back on a display device suitable for that purpose (e.g. an augmented reality device such as Microsoft®'s HoloLens®) while a laboratory worker repeats the same workflow or a modified version of it (e.g. with more or less customized parameters, work steps, etc.), for example to develop new methods of workflow. Of course, this can also be done iteratively over multiple cycles.

[0038] The invention is explained in more detail below with reference to embodiments shown in the drawings. [Brief description of the drawings]

[0039] [Figure 1] FIG. 1 is a schematic diagram of an exemplary apparatus configuration for performing a workflow, in one example a chemical workflow. [Diagram 2] FIG. 2 is a simplified principle diagram of an apparatus suitable for carrying out the method according to the invention. [Diagram 3] FIG. 3 is a block diagram of the basic steps of one embodiment of the recording and analysis method according to the present invention. [Figure 4] FIG. 4 is a block diagram of the detailed steps of one embodiment of the recording and analysis method according to the present invention. [Diagram 5] 5-7 show examples of screen displays of the recording and analysis method according to the present invention. [Figure 6] 5-7 show examples of screen displays of the recording and analysis method according to the present invention. [Figure 7] 5-7 show examples of screen displays of the recording and analysis method according to the present invention. [Figure 8] 8-9 are examples of screen display details of the recording and analysis method according to the present invention. [Figure 9] 8-9 are examples of screen display details of the recording and analysis method according to the present invention. [Figure 10]FIG. 10 is a simplified representation of the data sets displayed side-by-side in the form of tracks. [Figure 11] 11 and 12 show an example of grouped tracks. [Figure 12] 11 and 12 show an example of grouped tracks. [Figure 13] FIG. 13 shows an example of a screen display with workflow steps superimposed. [Figure 14] 14-15 show an example illustrating the retrieval of recorded parameters according to different criteria. [Figure 15] 14-15 show an example illustrating the retrieval of recorded parameters according to different criteria. [Figure 16] FIG. 16 shows an example illustrating the retrieval of recorded parameters from three different scientific experiment workflows. [Figure 17] FIG. 17 shows an example of a display of records resulting from multiple scientific experiment workflows. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0040] In the context of the present invention the following definitions are used:

[0041] A scientific experimental workflow is understood to be any kind of experiment, in particular of a chemical, biological or physical nature, for research and development purposes, for which the terms workflow or experiment are used herein.

[0042] Parameters are understood as any kind of measured and input variables, planned data and specified (target) values ​​that are relevant or interesting to the scientific experiment workflow. These include primarily planned and actual measured measurement variables such as temperature, stirring speed, weighing, etc., but also observation-based input by the user, e.g. in the form of text or voice input.

[0043] A data set includes image data created by individual video cameras during the workflow of a scientific experiment, and values ​​of individual parameters, i.e., measurements or user inputs, the image data and the parameter values ​​are assigned to a common reference time. The data set can also include planning data, such as target values ​​and workflow steps.

[0044] A track is to be understood as a display of a dataset or excerpts thereof, for example on a screen. These may for example be in image form (video recording, graphic display of measured values, etc.), but may also contain specific values ​​(specific measured values ​​at certain times, text, symbols, etc.) or may be of an acoustic nature (sound recordings, etc.). In particular, a track will be understood as a linear or columnar representation of the change in the value of a parameter over time.

[0045] A parameter event is understood as a situation in which a parameter has a particular state or value, or in which its state or value changes in a particular way.

[0046] According to Fig. 1, an exemplary workspace includes two reactors 1, 2, two heaters 3, 4 for the two reactors 1, 2, two temperature sensors 5, 6 for the two reactors 1, 2, two agitators 7, 8 for the two reactors 1, 2, two sensors 9 and 10 for the agitation speed of the two agitators 7, 8, a metering device 11 for the addition of chemicals to the reactors 1, 2, a metering device 12 for recording the amount effectively added, and video cameras 13, 14, 15. All these components are connected to a computer 16, which controls or delivers the data it captures to the computer 16. An input device 17 for manual input by a user and a microphone 18 for audio recording, e.g., voice input by a user, are also connected to the computer 16. The input device 17 may be, for example, the keyboard or mouse of the computer 16 as typically present, but may also be, for example, an input device in the form of a touch-sensitive screen (touch screen) or with visual or visual recording capabilities, for example an augmented reality device such as Microsoft's HoloLens (e.g., by gesture and / or voice control).

[0047] FIG. 2 shows the basic structure of an apparatus suitable for carrying out an embodiment of the method according to the invention, showing only those components relevant for the method according to the invention. The apparatus essentially comprises the already mentioned computer 16 and a screen (monitor) 19 connected thereto. Data sources are connected to the computer 16 on the input side. The data sources include, for example, the already mentioned video cameras 13, 14, 15, the sensors 5, 6, 9, 10 and the weighing scale 12. Depending on the nature of the scientific experiment workflow, further data sources may be provided, for example further sensors or other measuring devices. In FIG. 2, all these data sources are only symbolically represented by five blocks 21-25. The apparatus further comprises an input field 20 for operating and controlling the functions of the apparatus or the method. The input field may be realized by software, for example in the form of a menu structure dynamically superimposed on the screen 19 depending on the situation.

[0048] The computer 16 comprises, as its most important components, a digital data bank 30 and a program, symbolized by functional block 40, for reading, searching, calibrating, scaling, filtering, and visually displaying the data in the desired form on the screen 19.

[0049] The data delivered by the individual data sources during the workflow (video data, measurement data, acoustic data, manual input, etc.), i.e. the video recordings and the values ​​of the parameters relevant for the workflow, are stored in the data bank 30 in the form of individual, possibly hierarchically organized datasets, in each case temporally associated with a common reference time, which starts from the start of the workflow or is a standard time, e.g. Universal Time UTC or International Atomic Time TAI. The datasets symbolized by the five blocks 31 to 35 in Fig. 2 thus represent the synchronized evolution over time of the data delivered by the data sources. The reference time common to all datasets is symbolized in Fig. 2 by block 38.

[0050] A visual representation of the workflow takes place on a screen 19. The processed and parameterized data sets, and preferably also the individual work steps of the workflow, are symbolized in the form of five separate tracks 41, 42, 43, 44, 45 in FIG. 2 and displayed on the screen.

[0051] FIG. 3 illustrates in block diagram form one embodiment of the method according to the present invention.

[0052] In two introductory steps 701 and 702, the workflow is planned and prepared. This includes providing the work space, the necessary work equipment (reactors, metering devices, stirring devices, sensors, etc.) and the necessary materials, as well as video cameras and other input and recording devices.

[0053] In the next step 703, a workflow, in this example a chemical workflow, is executed, and image data from the video camera and values ​​of all parameters relevant for the workflow are recorded synchronously and stored in a data bank in the form of separate data sets. As already mentioned, the synchronization is performed with respect to a common reference time. The image data and the parameter values ​​are provided with a time stamp that can be generated automatically by a computer. 7030 shows the workflow, while blocks 7031, 7032, 7033, 7034 and 7035 show the recorded raw data of the video camera and the parameters including the user observations.

[0054] The values ​​of the parameters are recorded in a searchable format, i.e., it is possible to search the dataset for specific states, values, or events.

[0055] For simple measurements this is straightforward. However, image and / or sound recordings require analysis and processing. This is shown in step 704. For video camera image data, image analysis methods can be used that are able to autonomously recognize the desired image situation (e.g. color changes of the recorded object). For sound recordings, e.g. voice input by the user, speech recognition methods can be used. Finally, for text input, text recognition methods can be used. 7040 shows the workflow, with blocks 7041, 7042, 7043, 7044 and 7045 showing processing or preparation or parameterized data of the video camera and parameters (including user observations).

[0056] In a following step 705, the workflow is visually displayed on the screen 19, and the processed and parameterized data sets, and preferably also the individual work steps of the workflow, are displayed on the screen 19 in the form of separate tracks 7050, 7051, 7052, 7053, 7054, 7055.

[0057] The block diagram of FIG. 4 shows how the visual display of a workflow can be refined in a targeted manner, for example by displaying only certain data sets and / or only certain excerpts thereof depending on parameter events.

[0058] In step 706, all processed and parameterized data sets are visually displayed, similar to step 705 in Figure 3. In Figure 4 this is symbolized by blocks 7060, 7061, 7062, 7063, 7064 and 7065. In step 707, a parameter of interest is selected and displayed as a track 7072 in step 708. The selection of the parameter of interest can be made using the input field 20 (Figure 2).

[0059] In step 709, the desired parameter event for the selected parameter is entered and the event time at which the retrieved parameter event occurred is determined. Such a parameter event is a particular state (value) of the parameter in question, or a particular change in the parameter's state or value. However, the parameter event could also be text or voice entered by the user. Selection or entry of the parameter event of interest can be done using input field 20 (FIG. 2).

[0060] Finally, in step 710, time segments of the data set are visually displayed, which segments are focused on the previously determined event time points. That is, in the simplest case, only the video images and the values ​​of the parameters assigned to the event time points are displayed. Alternatively, and preferably, however, the changes in the video images and the values ​​of the parameters over a desired time frame or period are displayed. Such a period usually includes the event time point, for example the last 30 seconds leading up to the event time point. Advantageously, it is possible to set or select whether during playback the current time point is always displayed in the center of the display and the various tracks pass through the currently displayed time point or whether the tracks and the time axis are fixed and the current time point moves along the tracks. The selection or input of the period of interest can again be made by means of the input field 20 (Figure 2). The displayed tracks are symbolized in blocks 7100, 7101, 7102, 7103, 7102 and 7104 in Figure 4.

[0061] It is also possible to display only selected data sets or parameters in step 710. The selection of the parameters or video recordings to be displayed can also be made by means of an input field 20 (FIG. 2). As already mentioned, the input field 20 can advantageously be implemented in the form of a software menu displayable on the screen 19.

[0062] Fig. 5 shows, in a highly simplified manner, an exemplary screen layout of one embodiment of the method according to the invention. The screen shows an ongoing video image 101, in this example the process of recording how fluids are being metered by a metering device 103 into a first reactor 104 on an automatic blending device 102. A second reactor 105 is also evident from the video image 101. An information column 106 displays information about the camera used to record the video image 101. On top of the video image 101 relevant information is displayed in the form of an overlay graphic, including for example the name of the camera 107, the synchronized point in time 108 of the current recording, and their identifiers 109 and 110 to simplify the identification of the reactors being filmed.

[0063] The recording of the video images 101 is typically performed by a video camera installed at a fixed position or integrated into the device performing the workflow. However, it is also possible to use a mobile camera to record the video images 101, which may for example be supported and aligned by a suitable robotic device or a laboratory worker, or integrated into a portable device specially designed for augmented reality applications (such as Microsoft's HoloLens). Of course, it is possible and advantageous to use not only one recording source or camera, but also several (see also figures 7, 8 and 9). It is possible to point the cameras at different objects and / or to use different types of cameras (see also figure 9), for the same or different objects. The different types of cameras may for example differ in recording methods or wavelengths, such as infrared, ultraviolet, image intensification sensors, and sensors for radiation.

[0064] The screen layout shown in Fig. 5 furthermore shows a number of values ​​of selected parameters, here displayed in the form of grouped tracks 111 and 112 related to the two reactors 104 and 105. These tracks are advantageously designed, for example, to show the desired target values ​​of the parameters (here the reaction temperature T and the rotation speed v of the stirrer of the reactors) and the actual measured values ​​of the parameters as lines along a time axis 113, which is also displayed in the form of a track. In Fig. 5 the target values ​​are shown as solid lines and the actual values ​​as dotted lines. Advantageously, in addition the currently measured values ​​are superimposed (here the temperature T 45.0 ° C and the stirrer speed v 251 rpm (revolutions per minute) of the reactor # 1 and the temperature T 45.1 ° C and the stirrer speed v 248 rpm (revolutions per minute) of the reactor # 2).

[0065] The screen layout shows as the most important element a time axis 113 running parallel to the other tracks 111 and 112. The time axis indicates the exact point in time at which an ongoing recording or its playback is taking place. Here, the current point in time is shown for example as a solid line 114 crossing the various tracks. For example, a dashed line 115 as an auxiliary element can indicate a specific point in time (here, for example, every minute) and thus simplify the readout of parameter values ​​in the data tracks 111 and 112.

[0066] The screen layout shown in FIG. 6 shows a similar situation to FIG. 5, except that now a video image 201 is shown from an infrared (IR) camera 207, which displays cooler objects in darker colors and warmer objects in lighter colors. An information field 206 shows information about the camera used to record the video image 201. The information field 206 also includes a scale 2061 that shows in a simplified manner which color is assigned to which detected temperature. The exemplary image layout shows that the temperature of the first reactor 104 and its contents is hot, which is shown in a light color, while the temperature of the contents of the second reactor 105 is cold, which is shown in a dark color (shown in the dotted image). The temperature trace of the second reactor shows how the temperature of that reactor is lower (dashed temperature line) than the intended temperature (continuous temperature line). Furthermore, a warning message 2112 now superimposed shows that the temperature in the reactor is significantly different from the target value, indicating that the second reactor is clearly not heating up as planned and there may be a technical problem. Of course, the warning message 2112 may also include any appropriate text.

[0067] The exemplary screen layout shown in FIG. 7 shows a similar situation to FIGS. 5 and 6, except that in the video image 301 shown here, recordings from two video cameras are displayed partially overlapping. One recording comes from a video camera 107 operating in the visible spectral range, as shown in FIG. 5, and the other recording comes from an infrared camera 207, as shown in FIG. 6. Thus, in the video image 301, a video image, an infrared video image visible to the human eye and an infrared video image recording the thermal radiation of the object can be seen at the same time. Using known image processing programs, the entire IR image is not displayed overlaid, but only the parts of interest are overlaid. In FIG. 7, these excerpts show the thermal radiation of fluids 1041 and 1051 located in the two reactors 104 and 105. An information column 306 shows information about the camera used to record the video image 301.

[0068] Within the scope of the invention, it is also possible to display images from several, optionally different, video cameras, for example side-by-side or underneath one another, making several recordings visible, so that cameras operating in different spectral ranges can be directed at the same or different objects at the same time, the latter case being illustrated in figures 8 and 9, where the data and time tracks have been omitted for simplicity.

[0069] In Fig. 8 the screen layout is similar to Fig. 7, showing overlaid video images 301 of the first and second video cameras and video images 401 and 501 of two further video cameras, each directed towards one of the video cameras. Two reactors 104 and 105 are shown, showing in detail what is happening in the corresponding reactor. Now in Fig. 8 it is immediately clear that, for example, the second reactor 105 has a lower fluid level than the first reactor 104.

[0070] Advantageously, it is of course also possible to switch between different types of displays or camera inputs, i.e. the user can select whether he wishes to display, for example, the video image of the first video camera (as shown in FIG. 5), the IR image of the first IR camera (as shown in FIG. 6), an overlay of an excerpt of the IR camera on the first video camera (as shown in FIG. 7), or the entire IR image of the IR camera semi-transparently overlaid on the video image of the first video camera (not shown). It will be appreciated that a picture-in-picture display is also possible.

[0071] The screen layout shown in Fig. 9 shows a main video image 601 and two smaller sub-video images 6011 and 6012 within the main video image. The main video image shows the two reactors 104 and 105 as in Fig. 5, the sub-video image 6011 shows a video recording of only the first reactor 104 in the visible spectral range and the sub-video image 6012 shows a video recording of only the first reactor 104 in the infrared spectral range. The information section 606 has an integrated control field 607 that allows selecting or deselecting the display of the desired sub-video image. In this example, the sub-video image of the second reactor 105 is deselected.

[0072] It will be appreciated that visual and IR cameras are not the only input sources contemplated, but options include all commonly used and future conceivable recording methods, cameras, sensors (e.g., X-ray, among others), e.g., methods for identifying what is happening in a reactor without visually accessing the reactor. Further input options include, for example, UV radiation, image intensification cameras, ultrasonic sensors, etc. The following sections describe different ways in which further data tracks, not particularly from camera inputs, can be displayed in accordance with the present invention.

[0073] For example, Figure 10 shows several tracks from a simplified chemical workflow dataset. In this example, the workflow involves metering a powder into a stirred and heated container.

[0074] In the present example, the tracks run vertically across the screen parallel to the time axis 2000 and are arranged side by side. Track 2001 shows the recorded temperature profile in the cryostat. A second track 2002 shows the operation of the powder metering device and weighing scale. A third track 2003 shows the recorded rotation speed of the stirrer. A fourth track 2004 shows observations recorded by a user. Each track represents a parameter. Individual data points, e.g. measurements, and actions or observations are displayed at corresponding times along the time axis 2000.

[0075] In the example shown, it can be seen how, for example, immediately after the start of the workflow (time T0), first a target value of 50 °C for the cryostat is set, a target value of 450 rpm for the stirrer is set (time T0+10 s), and both devices are started (time T0+15 s). After that, the two devices start working and the measured parameters (cryostat temperature and stirrer speed) are recorded.

[0076] The time T0+38 s indicates the start of a weighing operation by the powder weighing device aimed at weighing 127 mg of a particular powder into a particular target container, which ends at the time T0+72 s and in which the fact that, as a result of e.g. the weighing scale input, 129 mg of powder has actually been added is stored.

[0077] A laboratory worker observing the automated operation observes the bubbling in the vessel at T0+40 seconds and T0+85 seconds and records the situation in the system. These observations are displayed in the track 2004. In FIG. 10, the observations are represented by symbols 2041 and 2042. In practice, the displayed observations are usually the corresponding text. Manual input of the observations is, for example, performed by the input device 17 shown in FIG. 1. Advantageously, however, such manual input is performed by a data processing device connected to the computer 16, in particular the use of a smartphone, a desktop, a laptop or a tablet computer is suitable, but the use of a device for augmented or virtual reality (for example Microsoft's HoloLens) is particularly advantageous. Conversely, these devices can also be used to display and process data and information recorded during the experiment.

[0078] Then, during the analysis phase of the workflow, lab workers can see exactly what happened at what point in time and what state each individual element was in at that point in time: for example, they can immediately see that at time T0+120 seconds the temperature of the cryostat was 25.7°C and the stirrer was rotating at 450 rpm.

[0079] Some apparatus or devices involved in the workflow comprise several features. For example, a cryostat has a pump, a heater and a temperature sensor. The values ​​or states of all these components can be recorded as parameters. In such cases, it is expedient to group the display of related tracks belonging to an element, as shown for example in FIG. 11 using the example of a cryostat. In a track group 2005 with the individual tracks 2051, 2052 and 2053, it is possible to implement a selection field 2055, which can preferably be opened by a corresponding button 2054, by which the individual track to be displayed can be selected (pump P, heater H, temperature T). In FIG. 11, the track of the temperature sensor of the cryostat is selected. Then, its individual track 2053 is displayed on the screen, as shown in FIG. 12. This simplifies the overview of the displayed data.

[0080] The information in the track can be displayed not only in textual or numerical form, but also in the form of graphs (curves) if desired, preferably with corresponding target or desired values ​​together with a corresponding scale. Preferably, a legend displayed therewith also assists in the identification of curves that differ, for example in color or line thickness / line shape. Figures 5 to 7 show examples of such graphical representations.

[0081] The workflow of a scientific experiment is often displayed as a workflow with defined symbols, in which in each case an inscribed arrow symbol constitutes a work step (or a group of combined work substeps). In the method according to the invention, such a workflow can be superimposed on a screen display of parameters and a video recording. Figure 13 shows such an example in a simplified form.

[0082] The screen layout of Fig. 13 largely corresponds to that of Fig. 5, except that it includes a video image 101, a grouped track 111 assigned to the reactor 104 and a time axis 113, and also shows the workflow track 3000. The grouped track 112 of Fig. 5 is not shown here. In the time axis 113, the times are entered as Universal Time UTC. A vertical line 114 indicates the current time.

[0083] The workflow shown in the workflow track 3000 corresponds to the planned workflow and indicates what should be done at what point in time. The individual work steps can be displayed as arrows and advantageously the status of the work steps can be symbolized by different colors. Already completed workflow steps 3001 can for example be shown in dark grey, currently ongoing workflow steps 3002 in medium grey and future workflow steps 3003 in light grey. Such a display showing a large amount of information, data, images etc. together with the video images provides for the first time a comprehensive log including all possible aspects of the scientific experiment workflow. Such a log, in combination with the ability to search for data within the entire recording, provides for the first time a complete, comprehensive and unprecedented ability to analyze and reuse the recorded workflow.

[0084] Such searches within a dataset are described in more detail below.

[0085] The various data sets are collated in a common data bank installed on the computer 16. Central to this is that all recordings and data sets (tracks, camera images, user inputs, etc.) are time-synchronized, so that events, values, parameters, etc. can be cross-referenced and searched. An important aspect of the invention is that all these different recordings and recorded data can be later combined and filtered according to the most diverse requirements, but also, in particular, searched. The search can include not only keywords (e.g. commentaries and observations added by laboratory workers), but also parameters and values ​​stored throughout the data sets. In addition, advantageously, visual tracks (or camera recordings) and sensor inputs can also be analyzed, interpreted and transformed by suitable devices and methods well known to those skilled in the art, and these inputs can also be included in the search. For example, the color of a formulation in a reactor is recognized by image recognition software, and the search displays all events, times, video sequences and images where the color of that formulation was recognized. Of course, this applies not only to what is observable by humans, but also to all possible physical and chemical parameters, data and values.

[0086] The dataset can be searched using search masks that allow searching for text or audio recordings, but also for specific values, variables, parameters, etc. All found entries can be listed in a table. Advantageously, the results can be further filtered and advantageously, by selecting a found entry, it is possible to jump directly to the point of the entry in the timeline and / or to mark or highlight the entry in the timeline and / or track.

[0087] For example, Figures 14 and 15 show how the recorded data and tracks can be searched according to the method according to the invention. Figure 14 shows a display advantageous for search purposes, which shows various tracks synchronized with a common time axis 4000, which in this case represents many other possible tracks (two video tracks 4001 and 4002 of two different video cameras, a track 4003 of a temperature sensor, a track 4004 of a powder weighing device, and an audio track 4005 with voice input by a laboratory operator). The video tracks 4001 and 4002 are displayed as individual images are shown along the time axis 4000, each image corresponding to a recording at the relevant time point.

[0088] FIG. 15 now illustrates how a search across the data tracks of a workflow operates according to the invention. A first exemplary query 4100 in a temperature sensor track 4003 or underlying dataset, for example, uses the search term "32°C", finds the first event 4101 where that value was measured by the sensor, and if necessary, displays the associated image (or recorded sequence) directly (in this example, time point 14:25:45). Furthermore, subsequent events of the same sensor track that also match the search term 32°C are displayed as well, and if necessary, events where that temperature was measured by other sensors. Of course, the totality of all found events can be filtered with reference to the required criteria. If more than one event (e.g. 32°C) is found, the user can select which associated time point or which associated video sequence he wants to display, since he knows, for example by experience, that the event is indeed that event, and the value sought may possibly just be an outlier or measurement error, or a measurement unrelated to the query. Based on his / her own experience, the user can easily determine, by referring to the associated video sequences and the additionally displayed parameter values, which of these search results actually represent an event of interest in the course of the workflow and which ones he / she wishes to investigate in more detail.

[0089] The second exemplary query 4200 searches the underlying data set of the acoustic track 4005, for example for acoustically recorded observations, for example for the term "bubbles". It is not only important here that the acoustic recordings are available in the form of a regular soundtrack, but also that the soundtracks are parameterized or digitized, for example by suitable speech recognition methods and thus parameterized or digitized for the respective speech recognition. When the user enters verbally, a text is also generated and stored at the corresponding time point, which can be searched using the search term. Thus, here the search query 4200 for "bubbles" also finds the event 4201 on the acoustic track 4005, the speech recognition recognizes the term "bubbles" in the underlying data set, and then displays in turn the desired other tracks, parameters, images and video sequences belonging to that time point. For multiple search results, the comments in the previous paragraph apply again.

[0090] The parameterization or digitization of raw data described above with respect to the acoustic tracks can of course be carried out for any desired tracks. The evaluation of sound recordings made in any conceivable way (video, IR, UV, acoustic, etc.) and their interpretation, parameterization and digitization are particularly advantageous. This is comparable, for example, to face recognition by means of a corresponding suitable software, whereby a camera recording of a face is provided with easily recognizable features, which can be stored in a comprehensive data bank, and the recorded images can be compared with each other and with images already available in the data bank and assigned. A similar digitization can also be used in the method according to the invention. Here (by way of example, not exhaustive) for example the color of a chemical preparation, its structure, the presence of inhomogeneities (particles, bubbles, etc.) can be recognized and digitized.

[0091] The search for data can of course not only be performed within a single workflow, but also advantageously across a number of (comparable) workflows, as shown very simply in Fig. 16. Here, across the three data sets of workflows 5001, 5002, 5003, in query 5100 a search is performed for, for example, the event "Temperature = 32°C", and in query 5200 a search is performed for a specific observation, for example the event "bubbles". Various events are found, which are represented by arrows 5101 and 5201 in Fig. 16. The found events are shown in tabular form in two tables 5102 and 5202. It can be seen that three events of "Temperature = 32°C" were found in two experiments, and two events of "bubbles" were found in two experiments. By selecting an event in the list, the laboratory worker can jump directly to the desired time point of the desired experiment and see the general conditions, parameters, etc. at that time. This allows for the first time a simple, comprehensive and fast search for parameters in a typical laboratory operation.

[0092] FIG. 17 shows an example display of multiple recordings 8001 and 8002 of a scientific experiment workflow performed and recorded at different times. Here, the user decides to select the addition of fluid to the test reactor as the currently displayed time point, for example by searching for "pH value ≧6.7" across the experiments of interest, and finds those events accordingly. These events can be displayed in a way that, for example, shifts the corresponding time axis or recordings to align with that common event "pH ≧6.7". Such a display provides a very intuitive way to find and visually compare events of interest related to the workflow, since all the parameters and recordings that the user desires are available.

[0093] The invention has been described using the example of a chemical workflow, however the invention is not limited to recording and analyzing chemical workflows but is also suitable for many other scientific experimental workflows.

Claims

1. 1. A method for recording a scientific experiment workflow, comprising: During the course of the workflow, a video recording is made containing a number of successive individual images and values ​​of at least a portion of the workspace in which the workflow is being executed; At the same time, all values ​​of a number of relevant parameters involved in the workflow are recorded, one of the parameters is a value measured by a temperature sensor; a digital data bank (30) storing separate data sets (31, 32, 33, 34, 35) for the video recording and for each parameter; the data sets (31, 32, 33, 34, 35) comprise the individual images of the video recording and values ​​of the parameters over time, assigned to a common reference time (38), so that at each time point there is a clear temporal relationship between the individual images of the video recording and the values ​​of the parameters; The data set of at least the parameters is stored in a searchable format, allowing for search for parameter events indicating searched values ​​or searched changes of the parameters. method.

2. 1. A method for recording and analyzing a scientific experiment workflow, comprising: During the course of the workflow, a video recording is made containing a number of successive individual images and values ​​of at least a portion of the workspace in which the workflow is being executed; At the same time, all values ​​of a number of relevant parameters involved in the workflow are recorded, one of the parameters is a value measured by a temperature sensor; a digital data bank (30) storing separate data sets (31, 32, 33, 34, 35) for the video recording and for each parameter; the data sets (31, 32, 33, 34, 35) comprise the individual images of the video recording and values ​​of the parameters over time, assigned to a common reference time (38), so that at each time point there is a clear temporal relationship between the individual images of the video recording and the values ​​of the parameters; storing said data set of at least the parameters in a searchable format, and searching for parameter events that indicate searched values ​​or searched changes of the parameters; and visually displaying images of the video recording and values ​​of at least the selected parameters present at the time of the retrieved parameter event or present during a period including the time of the retrieved parameter event, in temporal relation to one another. method.

3. The selected parameters are stored as a data set in the digital data bank, with target values ​​for those parameters, and are visually displayed along with the recorded actual values ​​of those parameters, and the changes over time are preferably in the form of a graph.

3. The method of claim 1 or 2.

4. said changes over time of said values ​​of at least selected parameters are visually displayed, if applicable, together with corresponding target values, in particular in the form of rows of numbers, in tracks arranged next to or above one another; 3. The method of claim 1 or 2.

5. A standard time is used as a common reference time, in particular Universal Time (UTC) or International Atomic Time (TAI), 3. The method of claim 1 or 2.

6. two or more video recordings of the workspace are made at different viewing angles and / or different wavelength ranges and stored as separate data sets; 3. The method of claim 1 or 2.

7. two or more video recordings of the workspace are made at different wavelength ranges and stored as separate data sets, and a display is generated from the data sets and visually displayed that is at least partially overlaid on the recorded workspace; 3. The method of claim 1 or 2.

8. said values ​​of at least selected parameters are entered into said digital data bank together with respective time points either automatically or manually by a user; 3. The method of claim 1 or 2.

9. Observations recorded by the user in text form, or preferably parameterized acoustic recordings, are stored as a separate dataset and / or a commentary is provided in the dataset; 3. The method of claim 1 or 2.

10. At least one video recording is made and is recorded in the visible spectral range; 3. The method of claim 1 or 2.

11. The video recording is carried out and recorded in the UV and / or IR spectral range; 3. The method of claim 1 or 2.

12. The method of claim 1 or 2, wherein the parameters include radiation values, such as radioactivity or X-ray radiation, magnetic field values, airflow values, or ultrasound measurements.

13. the video recording is searched for specific image content or variations thereof using image recognition methods; Preferably, the values ​​of at least selected said parameters are associated with an occurrence or change in the retrieved image content or with a selectable period of time before or after the occurrence of the retrieved image content or change therein; the images of the video recording are visually displayed in temporal relation to one another; 3. The method of claim 1 or 2.

14. the scientific experiment workflow planning data is stored as a dedicated dataset; 3. The method of claim 1 or 2.

15. two or more comparable scientific experiment workflows are recorded by at least one video recording of at least a portion of the workspace in which each workflow is being performed throughout the course of each workflow, while values ​​of parameters associated with each workflow are recorded and stored in a searchable format as separate data sets in a digital data bank, preferably displayed in a time-related manner; 3. The method of claim 1 or 2.

Images