Virtual pipetting

Abstract

The invention relates to virtual pipetting. The method for generating the control program of the laboratory automation equipment comprises the steps that configuration data of the laboratory automation equipment are received, and the configuration data encode the positions of components in the laboratory automation equipment; generating a three-dimensional model of the assembly of the laboratory automation equipment according to the configuration data, wherein the three-dimensional model further comprises a virtual pipette; displaying the three-dimensional model by using a virtual reality headset; receiving movement data of a motion sensing controller controlled by a user wearing the virtual reality headset, the movement data indicating three-dimensional movement of the motion sensing controller in space; determining movement of the virtual pipette in the three-dimensional model according to the movement data and updating the three-dimensional model according to the movement; a control program for the laboratory automation device is generated on the basis of the movement data, and the control program moves a pipette-bearing pipette arm of the laboratory automation device relative to the assembly on the basis of the movement of the virtual pipette in the three-dimensional model.

Description

[0001] This application is a divisional application of the Chinese invention patent application. The invention name of the original application is “Virtual Pipetting”, the application number of the original application is 201910123877.2, and the application date of the original application is February 19, 2019. Technical Field

[0002] The invention relates to a method, a computer program and a computer-readable medium for generating a control program for a laboratory automation device and to a control system. Background Art

[0003] Laboratory automation equipment is used to automate the tasks of laboratory assistants, such as testing patients for specific diseases. Typically, samples of blood, urine, feces, etc., are obtained and analyzed using biochemical procedures. These procedures involve various operations such as adding substances, incubation, and separation, as well as measurement processes that quantitatively or qualitatively measure the amount or presence of substances indicative of a specific disease.

[0004] Programming laboratory automation equipment is often a complex task requiring specialized skills. Programs for controlling laboratory automation equipment may need to be entered into a control PC. While graphical tools are available for generating such programs, more complex programs may need to be implemented using specialized scripting languages, which require specialized knowledge of programming and the operation of laboratory automation equipment. Consequently, laboratory assistants often perform the programs themselves with small numbers of samples.

[0005] To simplify the programming of laboratory automation systems with a robotic arm, it is known to manually move the robotic arm in a teach mode, in which the robotic arm can be freely moved to any position and a teach point can be set by simply pressing a button. The position of the robotic arm at the teach point is then saved and used to generate the control program.

[0006] EP 2 269 077 B1 relates to a graphical programming method in which parts of the layout of a laboratory automation device are displayed in a graphical interface. A pipette in the graphical interface can be moved for defining a control program for the laboratory automation device. Summary of the Invention

[0007] It is an object of the present invention to simplify the programming and / or control of laboratory automation equipment.

[0008] This object is achieved by the subject-matter of the independent claims. Further exemplary embodiments are evident from the dependent claims and the following description.

[0009] The first aspect of the present invention relates to a method for generating a control program for a laboratory automation device. The method can be performed using a system including a laboratory automation device, its controller, a virtual reality headset, and a motion sensing controller. Generally, the system can generate a three-dimensional virtual model of at least a part of the laboratory automation device and can display the model in the virtual reality headset. Using the motion sensing controller, a virtual pipette, which is also part of the three-dimensional model, can be manipulated. A person or user wearing the virtual reality headset can use the motion sensing controller to move the virtual pipette and / or aspirate and dispense virtual liquid within the model. The system can then record the manipulation of the virtual pipette and can generate a control program for the laboratory automation device, which actually repeats the procedure performed in the three-dimensional model.

[0010] In this way, the programming of the laboratory automation device can be simplified. The user can perform the task once using the virtual pipette within the virtual reality. The generated control program can be executed several times and can actually perform the task multiple times using the laboratory automation device.

[0011] According to an embodiment of the present invention, the method includes: receiving configuration data of the laboratory automation device, the configuration data encoding the positions of components in the laboratory automation device.

[0012] Generally, the laboratory automation device can be any device adapted to automatically perform the tasks of a laboratory assistant. At least such a laboratory automation device includes a pipetting arm, which is adapted to move a pipette between different positions and to aspirate and dispense liquid at these positions. The liquid can be aspirated from a chamber provided by a component and dispensed into a chamber provided by a component. Such components can include at least one of a reservoir, a sample tube, a microtiter plate, and / or a reagent container, etc.

[0013] The configuration data can encode the positions of the components. For example, for each component, the configuration data can include the type of the component and the component position and / or orientation provided in two-dimensional or three-dimensional coordinates.

[0014] The configuration data can be provided with a data file including information on the fixed layout of the laboratory automation device and / or with data generated by the laboratory automation device, which has sensors capable of determining which components are actually arranged in the laboratory automation device. For example, a laboratory assistant can arrange the components in the laboratory automation device and then can initiate the method. The laboratory automation device can then generate a configuration file, which encodes at least a part of the positions of the arranged components, for example using a barcode reader adapted to read barcodes on the components.

[0015] According to an embodiment of the present invention, the method further comprises: generating a three-dimensional virtual model of components of a laboratory automation device based on configuration data, the three-dimensional model further including a virtual pipette. Based on the configuration data, virtual components are generated in the three-dimensional model. The virtual components can be modeled as objects of an object-oriented programming language and / or can include a three-dimensional layout of the components. For example, the three-dimensional layout of the virtual components can be determined based on the type of components and the standard geometric layout of the components in the configuration data, and the standard geometric layout of the components can be stored, for example, in the device executing the method. Then, the standard layout can be moved by the position and / or orientation of the components in the configuration data.

[0016] It should be noted that the three-dimensional model does not include a model of the pipetting arm but includes a virtual pipette, which can replace the pipetting arm. The virtual pipette can be manipulated by a user viewing the three-dimensional model. Generally, the pipetting arm can move above the workbench of the laboratory automation device, and several components can be arranged on the workbench. For example, a cassette with sample tubes, one or more microtiter plates with reservoirs, and one or more containers with reagents can be provided on the workbench. In addition, carriers for microtiter plates, shakers, incubators, trays with disposable tips, etc. can be part of the laboratory automation device and / or be arranged on the workbench. All or some of these components can be encoded in the configuration data and / or provided in the three-dimensional model. The three-dimensional model can represent a part or the complete topography and / or layout of the working area of the laboratory automation device.

[0017] Generally, the three-dimensional model can include virtual components having cavities in which the tips of the virtual pipette can move. The three-dimensional model can include virtual components adapted to receive liquids.

[0018] According to an embodiment of the present invention, the method further comprises: displaying the three-dimensional model using a virtual reality headset. The virtual reality headset can be adapted to be worn by a user and / or to display a stereoscopic view of the three-dimensional model to the user. The virtual reality headset can generate two images for a person's eyes. The stereoscopic view of the three-dimensional model can depend on the position and / or orientation of the user's head. The virtual reality headset can include sensors for determining the position and / or orientation of the user's head. It must be noted that augmented reality can also be regarded as virtual reality.

[0019] According to an embodiment of the present invention, the method further comprises: receiving movement data of a motion-sensing controller controlled by a user wearing a virtual reality headset, the movement data indicating a three-dimensional movement of the motion-sensing controller in space; determining a movement of the virtual pipette in the three-dimensional model based on the movement data; and updating the three-dimensional model according to the movement of the virtual pipette.

[0020] The motion sensing controller may have sensors for determining the position and / or orientation of the motion sensing controller in space. For example, such sensors may include motion sensors, such as acceleration sensors. The user may hold the motion sensor with his or her hand and may move it in the view of the three-dimensional model. Then, the virtual pipette in the three-dimensional model is moved according to the sensed movement of the motion sensing controller. For example, the motion sensing controller may only include a handle and the pipette tip of the virtual pipette is arranged at the lower end of the handle in the three-dimensional model.

[0021] According to an embodiment of the present invention, the method further includes: generating a control program for the laboratory automation device according to the movement data, wherein the control program is adapted to move the pipetting arm of the laboratory automation device with a pipette relative to the component according to the movement of the virtual pipette in the three-dimensional model. It is possible to generate and / or execute the control program during the time when the virtual pipette is moved by the user. In other words, the control program can be generated and / or executed (almost) simultaneously with the movement of the virtual pipette. It is also possible that the control program is generated and / or executed when the user has completed the movement of the virtual pipette. It is also possible that the control program is generated and stored when the user has completed the movement of the virtual pipette, and the control program is executed at a later time.

[0022] For example, specific tracking or teaching points on the trajectory of the virtual pipette can be saved. Such points can be defined and / or recorded when the tip of the virtual pipette enters the cavity of the virtual component. Based on these points, the movement of the pipetting arm can be derived, and this movement also accesses all the points. It must be noted that the trajectory of the pipetting arm can be different from the trajectory of the virtual pipette.

[0023] The movement of the pipetting arm can be encoded into a control program, which when executed, performs the movement. The control program can be any data or data file, which when processed by the controller of the laboratory automation device, causes the corresponding movement of the pipetting arm. The control program can also be a script program that can be modified by the user.

[0024] According to an embodiment of the present invention, the method further includes: receiving activation data from the motion sensing controller, the activation data indicating the finger movement of the user on the motion sensing controller. Additionally, the thumb can be regarded as a finger. The motion sensing controller may include buttons, which can be adapted to sense whether it is pressed. The activation data can encode whether the button is pressed.

[0025] According to an embodiment of the present invention, the method further includes: determining the aspiration and / or discharge of virtual liquid in the three-dimensional model based on the position of the virtual pipette in the three-dimensional model, and activating the activation of the data indicating the motion sensing controller at this position. The motion sensing controller can operate like a mechanical and / or manual pipette. When the button is released, this can cause the virtual aspiration of the virtual pipette. When the button is pressed, this can cause the virtual discharge of the virtual pipette.

[0026] It is possible to encode in the three-dimensional model which type of virtual liquid is contained in each cavity of the virtual component and / or which cavities are empty. These types of liquids can also be encoded in the configuration data. The type of liquid in the configuration data can be determined based on the barcode on the corresponding component, which may have been read by a reader of the laboratory automation equipment. When the virtual pipette aspirates, it can be assumed that the virtual pipette contains the type of liquid in the cavity into which its tip enters. When the virtual pipette discharges, it can be assumed that the cavity into which the tip of the virtual pipette enters is filled with the type of liquid in the virtual pipette.

[0027] It must be noted that the type of liquid can refer to the chemical properties and / or content of the liquid, such as whether the liquid is a buffer solution, a sample, a reagent, etc., and / or refer to the physical properties of the liquid, such as viscosity, surface tension, density, etc.

[0028] According to an embodiment of the present invention, the control program is adapted to control the pipette of the pipetting arm to aspirate and discharge liquid according to the virtual pipette in the three-dimensional model. It can be recorded when the virtual pipette is aspirated and / or discharged. The generated control program can then also include commands for aspirating and / or discharging the pipette of the pipetting arm when moving the pipette or its tip in the corresponding cavity.

[0029] According to an embodiment of the present invention, the aspiration point and discharge point for the liquid are determined based on the movement data and activation data, wherein the movement of the pipetting arm is determined based on the aspiration point and discharge point. The point of discharging or expelling the liquid can be regarded as the most important tracking point and / or teachpoint. It is possible that the control program is generated only from these points. The movement of the pipetting arm between two consecutive points (such as the aspiration point and the subsequent discharge point) can be determined independently of the movement of the virtual pipette between these points. Therefore, the movement of the pipetting arm can follow a different path from the virtual pipette. The movement of the pipetting arm can be optimized more with respect to the path length and / or with respect to the limitations of the pipetting arm in view of possible movements and / or avoidance of collisions within the laboratory automation equipment.

[0030] According to an embodiment of the present invention, the movement of the liquid actually aspirated and discharged with a virtual pipette is performed in a three-dimensional model and / or displayed in a virtual reality headset. As already mentioned, the type of virtual liquid contained in the cavity of the virtual component can be stored and / or tracked. In addition, the presence of the liquid can be visually indicated in the three-dimensional mode. Different types of liquids can be displayed in different colors. For example, the cavities of virtual components such as sample tubes, reservoirs, reagent containers, etc. can be shown as filled with colored and / or transparent substances.

[0031] It is also possible that different types of pipettes and / or pipette tips are displayed together with differently colored virtual pipettes. The type of pipette tip can be defined by the maximum nominal volume of the liquid that can potentially be aspirated and / or by the geometry of the pipette tip.

[0032] According to an embodiment of the present invention, the assembly includes a disposable tip and the installation and movement of the virtual disposable tip are performed in a three-dimensional model and displayed in a virtual reality headset. The three-dimensional model can include a virtual component for the disposable tip. When moving the virtual pipette over such a virtual disposable tip, it can be modeled such that the virtual disposable tip is mounted on the virtual pipette. Additionally, the disposal of the virtual disposable tip can be modeled. The movement of the virtual disposable tip can be displayed for the user.

[0033] The control program determined according to the movement can then encode the installation and / or removal of the disposable tip at specific tracking points and / or teaching points.

[0034] According to an embodiment of the present invention, for each component, the configuration data encodes the type of the component and the position of the component in the laboratory automation device. The three-dimensional model can be generated from the modeling data encoding the geometric layout for each component. As already mentioned, the configuration data can include the type, position, and / or orientation of the component and can determine the three-dimensional layout of the virtual component according to the geometric layout, which can be associated with the type of the component. The geometric layout for the component can be pre-modeled and / or stored in the device executing the method. In other words, the three-dimensional model can be assembled according to the predefined geometric layout data of the component indicated in the configuration data.

[0035] According to an embodiment of the present invention, the type of liquid contained in the component is specified in the configuration data of the laboratory automation device. These types can be determined by sensors of the laboratory automation device and encoded in a configuration file. As already mentioned, barcodes on the component can encode the contained liquid. Then, different types of liquids can be visualized differently in the three-dimensional model, for example, in different colors.

[0036] It should be noted that other computer-readable codes, such as QR codes and / or RFID tags, can additionally be provided on the components and read by sensors of the laboratory automation device.

[0037] According to an embodiment of the present invention, the method further includes: manually arranging at least some of the components in the laboratory automation device; and determining at least some of the configuration data by using sensors of the laboratory automation device.

[0038] The laboratory automation device can include a fixed layout, that is, components that cannot be moved and / or removed by the user. Such components can include workbenches, carriers, shakers, etc. This fixed layout can be provided in the configuration data stored in the computing device that executes the method.

[0039] In addition, the laboratory automation device can include a variable layout, that is, components that can be moved, removed by the user, and included in the laboratory automation device. The positions and / or orientations of these components can be determined by using one or more sensors of the laboratory automation device. The configuration data thus determined can be sent from the laboratory automation device to the computing device that executes the method.

[0040] According to an embodiment of the present invention, movement data, activation data, and / or installation / removal of a virtual disposable pipette tip are recorded for one virtual sample. A user performing tasks in a three-dimensional model, that is, in virtual reality, can perform this task for only one sample, but the three-dimensional model can include multiple samples.

[0041] Then a control program can be generated such that it repeats the movement of the pipette, the aspiration and dispensing of the pipette, and the installation and removal of the disposable pipette tip for multiple real samples in the laboratory automation device. The control program can repeat these steps for each sample (the presence of which is determined by sensors) in the laboratory automation device. It is also possible that the control program can be changed by the user after its generation. For example, the user can include a control structure into the control program that repeats the steps performed in virtual reality.

[0042] It should be noted that tracking points or teaching points used to generate a partial control program can be defined relative to the cavity types typed by the virtual pipette. Such points can encode not only coordinate positions. For example, the tracking points or teaching points can encode "pipette in the sample tube", "pipette in the empty reservoir", etc. In this way, not only can the movement of the virtual pipette be repeated by the laboratory automation device, but also the steps of the tasks performed by the user for one virtual sample in virtual reality can be applied to several real samples in the laboratory automation device. In this case, it is not the exact movement of the virtual pipette but the movement scheme defined by the virtual pipette that can be repeated.

[0043] For example, the movement of the virtual pipette can be repeated for all samples of a microtiter plate. In this case, the movement can be determined by adding an offset to the original movement, the offset depending on the distance between reservoirs in the microtiter plate.

[0044] It is also possible that the movement of the virtual single-channel pipette is mapped to the movement of a real multi-channel pipette.

[0045] According to an embodiment of the invention, the motion sensing controller is adapted to be held in the hand of a user wearing a virtual reality headset. The motion sensing controller may include motion sensors, such as an acceleration sensor for generating movement data and / or may include buttons for generating activation data. It is possible that the motion sensing controller is designed to resemble the handle part of a real pipette. The pipette part may be modeled in a 3D model and / or may only exist in virtual reality.

[0046] According to an embodiment of the invention, the virtual reality headset includes two displays for generating a stereoscopic view of the 3D model. It is possible that the virtual reality headset is an augmented reality headset. It must be noted that the stereoscopic view of the 3D model can be displayed such that it is visible outside the real laboratory automation device. The user does not have to perform tasks inside the laboratory automation device.

[0047] According to an embodiment of the invention, the virtual pipette is a multi-channel pipette including a plurality of pipette tips. As mentioned above, the handle part of the pipette can be provided by the motion sensing controller. The pipette part having a plurality of pipette tips may only exist in the 3D model.

[0048] Another aspect of the invention relates to a computer program for generating a control program for a laboratory automation device, the control program being adapted to perform the steps of the methods described above and below when executed by a processor. The computer program can be executed in a computing device (such as a PC) communicatively interconnected with the laboratory automation device, the virtual reality headset, and the motion sensing controller.

[0049] Another aspect of the invention relates to a computer-readable medium in which such a computer program is stored. The computer-readable medium can be a floppy disk, a hard disk, a USB (Universal Serial Bus) storage device, a RAM (Random Access Memory), a ROM (Read-Only Memory), an EPROM (Erasable Programmable Read-Only Memory), or a FLASH memory. The computer-readable medium can also be a data communication network, such as the Internet, which allows downloading of the program code. Generally, the computer-readable medium can be a non-transitory or transitory medium.

[0050] Another aspect of the present invention relates to a control system for laboratory automation equipment, the system comprising laboratory automation equipment, a virtual reality headset, a motion sensing controller, and a computing device. The computing device is communicatively interconnected with the laboratory automation equipment, the virtual reality headset, and the motion sensing controller, for example via USB, Ethernet, and / or wireless connection. Further, the computing device is adapted to execute the methods described above and below.

[0051] The computing device may also be adapted to control the laboratory automation equipment and / or may also execute a control program generated by means of virtual reality.

[0052] It must be understood that the features of the methods described above and below may be features of the control system, computer program, and computer-readable medium described above and below, and vice versa.

[0053] Furthermore, the method may be a method for controlling laboratory automation equipment, which may then also execute the generated control program.

[0054] These and other aspects of the present invention will become apparent from the embodiments described below and will be elucidated with reference to the embodiments described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In the following, embodiments of the present invention will be described in more detail with reference to the drawings, in which

[0056] Figure 1 A control system according to an embodiment of the present invention is schematically shown.

[0057] Figure 2 A three-dimensional model used in a method for generating a control program according to an embodiment of the present invention is schematically shown.

[0058] Figure 3 A virtual pipette used in a method for generating a control program according to an embodiment of the present invention is schematically shown.

[0059] Figure 4 A flowchart of a method for generating a control program and for controlling laboratory automation equipment according to an embodiment of the present invention is shown.

[0060] The reference symbols used in the drawings and their meanings are listed in a list of reference symbols in a schematic form. In principle, the same parts in the figures are provided with the same reference symbols. DETAILED DESCRIPTION

[0061] Figure 1A control system 10 is shown, which includes laboratory automation equipment 12, a virtual reality headset 14, and a motion sensing controller 16. The control system 10 further includes a computing device 18, such as a PC, communicatively interconnected with the laboratory automation equipment 12, the virtual reality headset 14, and the motion sensing controller 16. Communication can be performed via Bluetooth© and / or Ethernet©.

[0062] The laboratory automation equipment 12 includes a workbench 20, to which a number of movable components 22 can be mounted. For example, the components 22 include a container 22a with a disposable pipette tip 24, a container 22b with a sample tube 26, a microtiter plate 22c with a reservoir 28, and a reagent container 22d. The reservoir 28 can contain a liquid 29.

[0063] The laboratory automation equipment 12 further includes a pipetting arm 30 having a pipette 32 and a sensor 34, which is adapted to determine the position and / or orientation of the component 22. The sensor 34 can also be and / or include a reader adapted to read a barcode or more generally a computer-readable code on the component 22. The sensor 34 can also be and / or can include a camera, a laser scanner, and / or any device adapted to determine the position and / or orientation of the component 22.

[0064] The virtual reality headset 14 includes two displays 36, which are adapted to provide two slightly different images to a user wearing the virtual reality headset 14. The two images can generate a stereoscopic view of the scene generated by the computing device 18, such that the user has a spatial impression of the scene. In addition, the virtual reality headset 14 can include a motion sensor 38, such as an acceleration sensor, which is adapted to determine the position and / or orientation of the user's head. In this way, the user's view position and / or view direction can be determined.

[0065] The motion sensing controller 16 is adapted to be held in the user's hand using a handle portion 40. Like the virtual reality headset 14, the motion sensing controller 16 includes a motion sensor, such as an acceleration sensor 42, which is adapted to determine the position and / or orientation of the motion sensing controller 16. The motion sensing controller 16 is used in the system to move a virtual pipette displayed in the virtual reality headset 14.

[0066] In addition, the motion sensor controller includes buttons 44, which can be pressed by the user. Using the buttons 44, aspiration and dispensing with the virtual pipette can be triggered.

[0067] Figure 1Also shown is data that can be exchanged between components during operation of system 10.

[0068] Initially, configuration data 46 from laboratory automation device 12 can be sent to computing device 18. Configuration data 46 can encode the position and / or orientation of at least some of the components 22 of laboratory automation device 12. For example, this data can be obtained using sensors 34.

[0069] Based on configuration data 46, computing device generates a three-dimensional or virtual model of at least a portion of laboratory automation device 12. A virtual pipette is also shown in this model.

[0070] Computing device 18 receives movement data 48 from virtual reality headset 14, and movement data 48 can encode the actual position and / or orientation of virtual reality headset 14.

[0071] Computing device 18 receives movement data and optional activation data 50 from motion sensing controller 16, and the movement and optional activation data 50 encodes the actual position and / or orientation of motion sensing controller 16 and the button press state.

[0072] Based on data 50, computing device 18 determines the position and / or orientation of the virtual pipette in the virtual model and generates display data 52 for virtual reality headset 14 by means of data 48.

[0073] In this way, as will be described in more detail below, computing device 18 is able to record the movement and optional activation of the virtual pipette during tasks performed by the user in the virtual model. When the user has completed the task, computing device 18 generates a control program 54 for laboratory automation device 12, which performs substantially the same task as laboratory automation device 12.

[0074] Finally, computing device 18 can execute the control program, which then generates control commands 56 that are sent to laboratory automation device 12 and / or controls pipetting arm 30 to repeat the task performed by the user with the virtual pipette.

[0075] Figure 2 Schematically shown is a three-dimensional model 58 that can be generated by computing device 18 and can be displayed in virtual reality headset 14. Three-dimensional model 58 can consist of virtual components 22' that have been generated by means of configuration data 46. The three-dimensional model can be regarded as a virtual model of at least a portion of the laboratory automation device.

[0076] For example, the virtual component 22' may include a virtual container 22a' with a virtual disposable pipette tip 24', a virtual container 22b' with a virtual sample tube 26', a virtual microtiter plate 22c' with a virtual reservoir 28', and a virtual reagent container 22d'.

[0077] The virtual component 22' may further include components that are based on the fixed real components of the laboratory automation device 12. For example, there may be a virtual workbench 20'.

[0078] It must be understood that the three-dimensional model 58 may consist of objects of a programming language and / or the virtual component 22' is entirely based on these objects. These objects can be transformed into display or image data 52 using a graphics rendering engine, and the display or image data 52 can then be displayed on the display 36.

[0079] In addition, the three-dimensional model 58 includes a virtual pipette 60. As Figure 2 shown, the virtual pipette 60 can be positioned in the model 58 such that when the virtual reality headset 14 is adapted to generate augmented reality, the user perceives the virtual pipette 60 as an extension of the motion sensing controller 16.

[0080] It is also possible that the three-dimensional model includes liquids 62, 64 as virtual components 22'. As an example, a virtual liquid 62 is shown in the virtual reservoir 28', and another virtual liquid 64 is shown in the virtual pipette. When these virtual liquids 62, 64 represent different types of liquids (such as samples and reagents), they can be colored differently.

[0081] Additionally, the virtual pipette 60 can be colored differently to indicate different types of pipette tips.

[0082] Figure 2 Also shown is the path 66 of the virtual pipette 60 in the three-dimensional model 58 and the tracking points 68a, 68b, 68c, 68d on the path 66, which can be recorded, for example, when the user presses the button 44.

[0083] Figure 3 Another virtual pipette 60' is shown, which is modeled as a multi-channel pipette 60' including a plurality of pipette tips 70. The methods described above and below are not limited to pipettes with one tip. It can also be performed with virtual pipettes having multiple tips (such as 8, 12, 96, 384 tips).

[0084] Figure 4A flowchart showing a method for generating a control program 54 for a laboratory automation device 12 and optionally for controlling the laboratory automation device 12 by executing the control program 54 is shown.

[0085] Initially, a user (such as a laboratory assistant) may manually arrange at least some of the components 22 in the laboratory automation device 12. For example, the user may arrange a container 22a with disposable tips 24, a container 22b with sample tubes 22b, a microtiter plate 22c, and a reagent container 22d on a workbench 20 (see Figure 1 ).

[0086] In step S10, the method can then be started by starting a corresponding computer program in the computing device 18.

[0087] The computing device 18 can then request a first part of the configuration data 46 from the laboratory automation device 12, which then determines the first part of the configuration data 46 using one or more sensors 34. For example, a laser scanner and / or a camera can determine the type, position, and / or orientation of the components 22. Additionally, a barcode scanner can scan barcodes on the components 22 and / or an RFID scanner can be used to determine their type and / or content.

[0088] For each component, the configuration data 46 can encode the type, position, orientation, and / or content of the component 22. It must be noted that the disposable tips 24, sample tubes 26, and reservoirs 28 can also be considered components 22, which are sub-components of the components 22a to 22c. For example, the configuration data 46 can be provided in a tree structure that describes the arrangement of the components 22 within other components 22.

[0089] It is also possible that the configuration data 46 specifies the type of liquid contained in the component 22. This type can be the content or can be derived from the content of the corresponding component 22.

[0090] The first part of the configuration data 46 related to the components 22 that can be arranged by the user can be regarded as the configuration data for the variable layout of the laboratory automation device. A second part of the configuration data 46' related to the fixed components (i.e., fixed layout) of the laboratory automation device 12 that cannot be removed or moved by the user can be directly stored in the computing device 18. However, this part can also be sent from the laboratory automation device 12 to the computing device 18. For example, the workbench 20 can be encoded as a fixed component in the configuration data 46'.

[0091] In step S12, configuration data 46, 46' of the laboratory automation device 12 is received in the computing device 18, and the computing device 18 then generates a three-dimensional model 58 of the components 22 of the laboratory automation device 12 based on the configuration data 46, 46'.

[0092] In the computing device 18, modeling data 72 can be stored, and the modeling data 72 encodes the geometric layout for the corresponding components 22 for each type of component 22. The modeling data 72 can encode the coordinates, faces, line models, etc. of the corresponding geometric layout.

[0093] Using the modeling data 72, the computing device 18 can generate the three-dimensional model 58 by moving and orienting the corresponding modeling data 72 by utilizing the positions and orientations encoded in the configuration data 46, 46'.

[0094] The three-dimensional model 58 additionally includes a virtual pipette 60. Additionally for this virtual pipette 60, the modeling data can be stored in the computing device 18.

[0095] The user can now put on the virtual reality headset 14 and can pick up the motion sensing controller 16.

[0096] In step S14, the computing device 18 receives movement data 48 from the virtual reality headset 14 and determines the position and orientation of the user's head. Thereby, the view direction and the field of view can be determined. Using the field of view and the view direction, the computing device can render a corresponding scene for each eye of the user and / or can generate display data 52. The display data 52 can be generated using a rendering engine. Then when displaying the corresponding view of the three-dimensional model 58, the display data 52 is then sent to the virtual reality headset 14, for example as Figure 2 shown.

[0097] The user now sees at least a part of the laboratory automation device 12 in virtual reality. In particular, he or she sees virtual components 20', 22', 24', 26', etc. that may look similar to the corresponding real components 20, 22, 24, 26, etc. The modeling data 72 can be provided such that the virtual components may look the same or similar to their real counterparts.

[0098] As already mentioned, different types of virtual liquids 62, 64 can be visualized differently in the three-dimensional model, for example using different colors. This can also be the case when the corresponding real liquids look rather similar. This can help the user better distinguish the virtual liquids 62, 64 from each other. It is also possible that when the virtual liquid 64 is added to the other liquid 62 that already exists, a new liquid corresponding to the mixture is associated with the corresponding component 22', and this new liquid can be visualized with another different color.

[0099] In step S14, computing device 18 also receives movement and activation data 50 from motion sensing controller 16. Movement data 50 indicates the three-dimensional movement of motion sensing controller 16 in space and computing device 18 can determine the position and / or orientation of virtual pipette 60 in three-dimensional model 58. A view of three-dimensional model 58 is displayed to the user together with virtual pipette 60, and the user sees virtual pipette 60 that he moves through three-dimensional model 58.

[0100] Generally, the movement of virtual pipette 60 in three-dimensional model 58 is determined based on movement data 50, and three-dimensional model 58 is updated according to the movement of virtual pipette 60.

[0101] Computing device 18 can record the path 66 of one or more tips of the virtual pipette. This path 66 can later be used to generate control program 54. It is also possible that path 66 is displayed in three-dimensional model 58. This can help the user verify whether he has performed the task correctly.

[0102] Activation data 50 from motion sensing controller 16 can indicate finger presses and / or finger movements of the user on button 44 of motion sensing controller 16. Whenever the user presses button 44, it can be assumed that virtual pipette 60 sucks in liquid 62, 64 when its tip is within virtual liquids 62, 64. Accordingly, it can be assumed that when the button is released, the liquid 62, 64 in pipette 60 is discharged into virtual component 22' where the tip of pipette 60 is located. However, activation data 50 can be evaluated in another way. For example, a short button press can cause suction or discharge regardless of whether virtual pipette 60 is filled.

[0103] Generally, the suction and / or discharge of virtual liquids 62, 64 in three-dimensional model 58 is determined according to the position of virtual pipette 60 in three-dimensional model 58 where activation data 50 indicates the activation of motion sensing controller 16.

[0104] It must be noted that three-dimensional model 58 can also show virtual liquids 62, 64 in virtual pipette 60. In this way, when virtual pipette 60 filled with liquid 62, 64 is moved, the movement of the liquid actually sucked in and discharged with virtual pipette 60 can be performed in three-dimensional model 58 and can be displayed in virtual reality headset 14.

[0105] In addition, it is possible that virtual component 22' includes virtual disposable tip 24'. In this case, the installation and movement of disposable tip 24 can be performed in three-dimensional model 58 and can be displayed in virtual reality headset 14.

[0106] For example, when the mounting portion of the virtual pipette 60 is located at the position of the virtual disposable tip 24', it can be determined that the virtual disposable tip 24' is mounted to the virtual pipette 60. Corresponding activation data 50 (such as a double click of the button 44) can indicate the removal of the virtual disposable tip 24'.

[0107] The computing device 18 can additionally record the tracking points 68a, 68b, 68c, 68d on the path 66. Whenever a specific event occurs (such as the installation or removal of the disposable tip 24' (tracking point 68a, see Figure 2 ), the aspiration of the liquids 62, 64 (tracking points 68b, 68d), the dispensing of the liquids 62, 64 (tracking point 68c)), such tracking points can be recorded. The tracking point 68a can be the installation point, the tracking points 68b, 68d can be the aspiration points and the tracking point 68c can be the dispensing point.

[0108] Generally speaking, an event occurs when the content and / or configuration of the virtual pipette 60 changes. It is also possible that an event occurs when the content of the virtual component 22' changes.

[0109] The tracking points 68a, 68b, 68c, 68d can encode the location and / or the type of the event. In the case of an event occurring, the location can be encoded as three-dimensional coordinates and / or encoded as the component 22'. The type of the event can be encoded with the types of the liquids 62, 64 aspirated or dispensed using the virtual pipette.

[0110] The tracking points 68a, 68b, 68c, 68d can be used to generate the control program 54. It is also possible that the tracking points are displayed in the three-dimensional model 58. This can additionally help the user verify whether he has correctly performed the task.

[0111] When the user has completed his or her task in the virtual reality, he can remove the virtual reality headset 14 and can command the computing device 18 (or the corresponding computer program running in the computing device 18) to generate the control program 54.

[0112] In step S16, then, based on the movement and activation data 50 and in particular based on the information derived therefrom (such as the path 66 and / or the tracking points 68a, 68b, 68c, 68d), a control program 54 for the laboratory automation device 12 is generated.

[0113] Generally, the control program 54 is generated such that it is adapted to move the pipetting arm 30 with the pipette 32 of the laboratory automation device 12 relative to the component 22 according to the movement of the virtual pipette 60 in the three-dimensional model 58, for controlling the pipette 32 of the pipetting arm 30 to aspirate and dispense liquid according to the virtual pipette 60 in the three-dimensional model 58 and / or to mount and dismount disposable tips 24 according to the virtual pipette 60 in the three-dimensional model 58.

[0114] In one example, the control program 54 can be generated such that, via the pipetting arm 30 with the pipette 32, only the aspiration and dispensing of the virtual pipette 60, optionally the mounting and dismounting of the disposable tip 24, and the movement to the corresponding position are repeated. The control program does not have to know the content within the corresponding component and / or the liquid.

[0115] In another example, the user performs his or her task for a virtual sample 26', i.e., records movement and activation data 50 for a virtual sample and generates the control program 54 such that it repeats the movement of the pipette 32 arm and / or the aspiration and dispensing of the pipette 32 for a plurality of real samples 26 in the laboratory automation device 12. For example, this can be achieved by simply moving the positions where the aspiration and dispensing of the pipette 32 and optionally the mounting and dismounting of the disposable tip occur to the next adjacent position at the corresponding component 22.

[0116] In another example, the movement of the pipetting arm 30 and the pipette 32, the aspiration and dispensing of the pipette 32, and / or optionally the mounting and dismounting of the disposable tip 24 are determined according to tracking points 68a, 68b, 68c, 68d. According to the events associated with the tracking points, the corresponding commands for the control program 54 can be derived. For example, the tracking point 68a can be converted into the command "mount removable tip" and the tracking point 68a can be converted into "towards sample", etc.

[0117] Also in this case, the control program 54 can be generated such that it repeats the tasks for several samples. For example, it is possible that the configuration data 46 can contain information about a plurality of actually existing liquids 29 and a task has been performed for one of the virtual liquids 62. Then the control program 54 can be generated to perform the task for all samples 26.

[0118] In step S18, optionally, the control program 54 can be modified into a control program 54'. For example, the generated control program 54 can be a script, which can be further modified by the user. For example, a loop control structure can be inserted into the control program 54 by the user. It is also possible that additional steps (such as incubation of samples) are included in the control program 54.

[0119] In step S20, for example, when a user commands a computer program in a computing device to execute control programs 54, 54', the control program 54 or 54' is executed by the computing device 18. This can be done several times. For example, when the control programs 54, 54' are completed, the user can arrange new components 22 in the laboratory automation device 12 in the same layout and can start the control programs 54, 54' again.

[0120] When the control programs 54, 54' are executed, control commands 56 are generated and the laboratory automation device 12 executes the tasks that have been designed by the user in virtual reality. In particular, the same tasks executed for one sample 26' in virtual reality can be executed multiple times for multiple samples 26 by the laboratory automation device 12.

[0121] Although the present invention has been illustrated and described in detail in the drawings and the foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the present invention is not limited to the disclosed embodiments. From the study of the drawings, the disclosure, and the appended claims, other variations of the disclosed embodiments can be understood and effected by those skilled in the art who are experienced in the art and practice the claimed invention. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or controller or other unit may perform the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.

[0122] List of reference numerals

[0123] 10 Control system

[0124] 12 Laboratory automation device

[0125] 14 Virtual reality headset

[0126] 16 Motion sensing controller

[0127] 18 Computing device

[0128] 20 Workbench

[0129] 22 Component

[0130] 22a Container

[0131] 22b Container

[0132] 22c Microtiter plate

[0133] 22d Reagent container

[0134] 24 disposable pipette tips

[0135] 26 sample tubes

[0136] 28 reservoirs

[0137] 29 liquids

[0138] 30 pipetting arms

[0139] 32 pipettes

[0140] 34 sensors

[0141] 36 displays

[0142] 38 motion sensors

[0143] 40 handle parts

[0144] 42 motion sensors

[0145] 44 buttons

[0146] 46 configuration data

[0147] 46’ configuration data

[0148] 48 movement data

[0149] 50 movement and activation data

[0150] 52 display data

[0151] 54 control programs

[0152] 54’ modified control programs

[0153] 56 control commands

[0154] 58 three-dimensional models

[0155] 20’ virtual workbenches

[0156] 22’ virtual components

[0157] 22a’ virtual containers

[0158] 22b’ virtual containers

[0159] 22c’ virtual microtiter plates

[0160] 22d’ virtual reagent containers

[0161] 24’ virtual disposable pipette tips

[0162] 26’ virtual sample tubes

[0163] 28’ virtual reservoirs

[0164] 60 virtual pipettes

[0165] 62 virtual liquid

[0166] 64 virtual liquid

[0167] 66 path

[0168] 68a tracking point

[0169] 68b tracking point

[0170] 68c tracking point

[0171] 68d tracking point

[0172] 60’ multi-channel virtual pipette

[0173] 70 pipette tip

[0174] 72 modeling data.

Claims

1. A method for generating a control program (54) for a laboratory automation device (12), the method comprising: Receiving configuration data (46) of the laboratory automation device (12), the configuration data (46) encoding the position and / or orientation of components (22) in the laboratory automation device (12), wherein at least some of the configuration data (46) is determined using one or more sensors (34) of the laboratory automation device (12), wherein the laboratory automation device (12) includes a variable layout having components (22) that are user - movable, removable, and includable in the laboratory automation device (12), and wherein the position and / or orientation of these components (22) are determined by the one or more sensors (34) of the laboratory automation device (12); Generating a three - dimensional model (58) of the components (22) of the laboratory automation device (12) according to the configuration data (46), the three - dimensional model (22) further including a virtual pipette (60); Displaying the three - dimensional model (58) using a virtual reality headset (14); Receiving movement data (50) of a motion - sensing controller (16) controlled by a user wearing the virtual reality headset (14), the movement data (50) indicating three - dimensional movement of the motion - sensing controller (16) in space; Determining the movement of the virtual pipette (60) in the three - dimensional model (58) according to the movement data (50) and updating the three - dimensional model (58) according to the movement of the virtual pipette (60); Generating a control program (54) for the laboratory automation device (12) according to the movement data (50), wherein the control program (54) is adapted to move a pipetting arm (30) with a pipette (32) of the laboratory automation device (12) relative to the components (22) according to the movement of the virtual pipette (60) in the three - dimensional model (58).

2. The method according to claim 1, further comprising: Receiving activation data (50) from the motion - sensing controller (16), the activation data (50) indicating finger movement of the user on the motion - sensing controller (16); Determining the aspiration and / or discharge of liquid (62, 64) in the three - dimensional model (58) according to the position of the virtual pipette (60) in the three - dimensional model (58), wherein at that position the activation data (50) indicates activation of the motion - sensing controller (16); wherein the control program (54) is adapted to control the pipette (32) of the pipetting arm (30) to aspirate and discharge liquid according to the virtual pipette (60) in the three - dimensional model (58).

3. The method according to claim 2, Among them, The aspiration points (68a, 68c) and discharge point (68c) for the liquid (62, 64) are determined according to the movement data and activation data (50); Among them, the movement of the pipetting arm (30) is determined according to the suction points (68a, 68c) and the discharge point (68c).

4. The method according to claim 2 or 3, Among them, The movement of the liquid (62, 64) virtually aspirated and discharged by the virtual pipette (60) is performed in the three-dimensional model (58) and displayed in the virtual reality headset (14).

5. The method according to any one of claims 2 to 4, Among them, The assembly (22) includes a disposable tip (24), and the installation and movement of the disposable tip are performed in the three-dimensional model (58) and displayed in the virtual reality headset (14).

6. The method according to any one of the preceding claims, Among them, For each assembly (22), the configuration data (46) encodes the type of the assembly (22) in the laboratory automation device (12) and the position of the assembly (22); Among them, the three-dimensional model (58) is generated according to the modeling data (72) encoding the geometric layout of each assembly (22).

7. The method according to any one of the preceding claims, Among them, The assembly (22) includes at least one of the following: a reservoir (28), a microtiter plate (22c), a reagent container (22d), and a sample tube (26).

8. The method according to any one of the preceding claims, Among them, The type of the liquid (62, 64) contained in the assembly (22) is specified in the configuration data (46) of the laboratory automation device (12); Among them, different types of liquids (62, 64) are visualized differently in the three-dimensional model (58).

9. The method according to any one of the preceding claims, Among them, Movement data and / or activation data (50) are recorded for a virtual sample; Among them, the control program (54) is generated such that it repeats the movement of the pipetting arm (30) and / or the aspiration and discharge of the pipette (32) for a plurality of real samples in the laboratory automation device (12).

10. The method according to any one of the preceding claims, Among them, The motion sensing controller (16) is adapted to be held in the hand of a user wearing the virtual reality headset (14); Among them, the motion sensing controller (16) includes a motion sensor (42) for generating the movement data (50); Among them, the motion sensing controller (16) includes a button (44) for generating activation data (50).

11. The method according to any one of the preceding claims, Among them, The virtual pipette (60’) is a multi-channel pipette including a plurality of pipette tips (70).

12. A computer program for generating a control program (54) for a laboratory automation device (12), which when executed by a processor, is adapted to perform the steps of the method according to any one of the preceding claims.

13. A computer-readable medium storing the computer program according to claim 12.

14. A control system (10) for a laboratory automation device (12), the system comprising: the laboratory automation device (12); The control system (10) is characterized in that it further comprises: a virtual reality headset (14); a motion sensing controller (16); a computing device (18), the computing device (18) being communicatively interconnected with the laboratory automation device (12), the virtual reality headset (14) and the motion sensing controller (16) and being adapted to execute the method according to one of claims 1 to 11.

Images