Laboratory automation device with transparent display in door

Abstract

Laboratory automation apparatus (10) comprising a workspace (26) with liquid containers (16) and a pipetting arm (20) for moving liquids between the liquid containers (16), an enclosure (28) enclosing the workspace (26), a door (30) of the enclosure (28) to access the workspace (26), wherein the door (30) comprises a transparent display (32) for displaying information (34) and allowing a person to see into the workspace (26), and a tracking sensor (36) to track an eye position (46) of a person (42). The laboratory automation apparatus (10) is adapted to determine the eye position (46) of the person (42) from sensor data acquired by the tracking sensor (36) and to display information (34b) for a notification area (48) in the workspace (26) on the transparent display (32) such that the information (34b) is displayed from the perspective of the person (42) in front of the notification area (48).

Description

TECHNICAL FIELD

[0001] The present invention relates to a laboratory automation device. Furthermore, the present invention relates to a method, a computer program, a computer readable medium and a controller for operating a laboratory automation device. BACKGROUND

[0002] Laboratory automation devices are used to automate tasks of laboratory assistants, e.g. for testing a patient for a certain disease. Typically, a sample of the patient's blood, urine, feces, etc. is collected and analyzed by biochemical procedures. Such procedures comprise various operations like adding substances, incubation, separation, etc. and measuring processes for quantitatively or qualitatively measuring the amount of a substance indicative of a certain disease, the presence or absence of a substance indicative of a certain disease.

[0003] A certain laboratory automation device comprises a workspace which is enclosed by a housing with a door. Inside the housing reagents contained in containers can be handled with a pipetting arm which can aspirate liquid from one container and can dispense liquid into another container. The door can be transparent to enable a person to see inside the workspace, however, the door can shield from unintentional access to the workspace.

[0004] The laboratory automation device can be connected with a controller, typically a PC, which e.g. performs the control of the pipetting arm. Status and error messages of various components inside the laboratory automation device can be displayed on a monitor of the PC. A schematic representation of the workspace based on graphical symbols can be shown on the monitor. In order to check the status of the laboratory automation device, a person has to look at the monitor and has to identify which component inside the workspace is related to a certain message and / or symbol on the monitor. SUMMARY

[0005] It is an object of the present invention to simplify the supervision of a laboratory automation device.

[0006] This object is achieved by the subject matter of the independent claims. Further exemplary embodiments are evident from the dependent claims and the following description.

[0007] A first aspect of the present invention relates to a laboratory automation device. Generally, the laboratory automation device can be any device suitable for automatically performing tasks of a laboratory assistant. At least such a laboratory automation device comprises a pipetting arm which is suitable for moving a robotic pipette between different positions and for aspirating and ejecting liquid at these positions. Additionally or alternatively, such a laboratory automation device comprises at least a gripper arm which is suitable for moving a robotic gripper between different positions and for removing objects from these positions or placing objects to these positions. Liquid can be aspirated and ejected from cavities provided by liquid containers which can be positioned on a worktable of the laboratory automation device. Such liquid containers can comprise at least one of a well, a sample tube, a microtiter plate and / or a reagent container.

[0008] According to an embodiment of the present invention, the laboratory automation device comprises a work space with liquid containers and a pipetting arm for moving liquid between the liquid containers, an enclosure enclosing the work space, a door of the enclosure to access the work space, wherein the door comprises a transparent display for displaying information and allowing a person to see into the work space, and a tracking sensor to track an eye position of the person.

[0009] With such an apparatus, the laboratory automation device is suitable for displaying information directly on the door through which the person can see into the interior of the enclosure, in particular the work space. Furthermore, the laboratory automation device is suitable for determining which area of the interior of the work space the person is looking at. In this way, the person can best be informed about procedures and errors inside the laboratory automation device.

[0010] The enclosure and the door can completely enclose the work space. However, it is also possible that the access to the work space is restricted from certain directions only by the enclosure.

[0011] The transparent display can be an OLED or an LCD display. This type of display can display information, text and graphics on a transparent screen.

[0012] According to an embodiment of the present invention, the laboratory automation device is suitable for determining the eye position of the person from sensor data acquired by the tracking sensor and for displaying information for a notification area in the work space on the transparent display such that the information is displayed from the perspective of the person in front of the notification area. These steps can be controlled by a controller of the laboratory automation device which can be a computing device and / or a PC connected to the laboratory automation device. The controller can be an external device or can be comprised in the laboratory automation device and in particular in its enclosure. The pipetting arm, the tracking sensor and the transparent display can be connected for data communication with the controller.

[0013] A transparent display can be used to display an augmented reality application that augments a work space of a laboratory automation device. Such an augmented reality application can be executed in a controller. A person's perspective can be defined by her or his eye position. From the eye position, it can be determined which part of the transparent display is in front of a notification area. This part can define a display area in which information is displayed.

[0014] The information shown on the transparent display can be a notification and / or an error message related to a specific object / component in the work space. The augmented reality application can display the information in the vicinity and / or above the respective component with respect to the person's eye position.

[0015] The information can comprise a symbol indicator, which can be colored and / or blinking. The indicator can be an outline of the notification area.

[0016] The eye position can be set as a three-dimensional coordinate with respect to a coordinate system defined by the laboratory automation device.

[0017] According to embodiments of the present invention, the notification area is projected onto the transparent display towards the eye position. By the projection, a display area on the transparent display is determined in which the information shall be displayed. It is possible that the laboratory automation device and / or its controller has an internal model of the current setup of the work space. In particular, the positions and optional types of components inside the work space can be stored in the internal model. Such components can include liquid containers, other types of containers such as pipette tip containers, test samples, pipetting arms, etc. It is possible that three-dimensional outlines of the components are stored with the components. Such outlines can be determined from the type of the components.

[0018] As the laboratory automation device can have to determine the movements of the pipetting arms and grippers between these components, it is possible that an internal model of the work space already exists, which can include the positions and / or extensions of the components in the work space. Furthermore, such an internal model can be generated in a background work table recognition, in which the laboratory automation device determines the content of the work space by sensors.

[0019] This specific shape of the component or more generally a box shape or other general shape at the component position can be projected onto the transparent display towards the person's eye position. This determines that from the person's perspective, the display area is in front of the component. When showing information in this display area, the information is directly related to the component.

[0020] According to embodiments of the application, the notification area is an area in which a component of the laboratory automation device is located. For example, the information displayed on the transparent display is status information of the component. The information can indicate that there is too little pipette tip and / or the liquid content of the liquid container is too low when refilling the container. The status information can be an error message. The status information can be a request to interact with the corresponding component in the work space. It can be that the component is removable and the information indicates that the component has to be removed.

[0021] It can be that the work space is monitored by a camera or more generally a sensor. By evaluating the sensor data, it can be determined whether a component is in the correct position. When a component is in the wrong position, a corresponding error message is generated.

[0022] According to embodiments of the application, the notification area is an area in which a component of the laboratory automation device is to be placed. In general, the notification area can also be a blank space of the work space. The information displayed on the transparent display can be status information of the component to be placed in the notification area. The information can indicate that a liquid container has to be put into the area.

[0023] According to embodiments of the application, the tracking sensor comprises one or more cameras. The eye position can be estimated from the video stream. For example, the position and / or size of the head of the person can be extracted by extracting the face of the person from the video stream.

[0024] According to embodiments of the application, the tracking sensor comprises an eye tracking sensor. Such a sensor can comprise an infrared camera and an infrared light source, the infrared light being reflected from the eyes of the person. Using an eye tracking sensor can more accurately estimate the viewing direction and the eye position compared to using only a video stream.

[0025] According to embodiments of the application, the tracking sensor is adapted to determine that more than one person is in its sensing area. For example, it can be determined from the video stream that a person is located in front of the door and / or is looking through the door. Such a video stream can be provided by the camera of the tracking sensor.

[0026] According to embodiments of the application, the person whose eye position is to be determined is selected via gesture detection performed by the tracking sensor from sensor data of the tracking sensor. The video stream of the camera can also be used for gesture detection. There can be a gesture, such as waving a hand, which indicates that the person making the gesture is the person for which the eye position is to be determined and / or for which the augmented reality is to be generated.

[0027] It can also be that, when more than one person is detected by the tracking sensor, the information is displayed without augmented reality features.

[0028] According to embodiments of the present invention, a person whose eye position is to be determined is selected via a transparent display. The transparent display can also comprise a touch screen. The transparent display can display pictures of a group of users that can be taken by a camera. The picture of the person whose display has the correct perspective for the augmented reality can be touched on the transparent display to select the corresponding person.

[0029] According to embodiments of the present invention, a door has a position sensor for determining the position of the door. The door can be a slidable door or a hinged door. There can be a "closed", a "bench accessible" and a "fully open" position. The position sensor can also be used to stop a pipetting arm when the door is opened and / or moved to a certain position.

[0030] According to embodiments of the present invention, a display area on the transparent display where information is displayed depends on the position of the door. There can be several positions of the door in which a person can see into the work space through the door.

[0031] By means of the position sensor, two or more positions of the door can be detected, such as open and closed. A three-dimensional model of the door and / or the transparent display together with its position and / or orientation can be stored in the controller. This model can be used to determine the projection of the notification area onto the transparent display. The position of the door can additionally be taken into account for determining the projection by means of the position sensor and / or the detected position of the door.

[0032] According to embodiments of the present invention, the transparent display comprises a touch screen. As mentioned above, the transparent display can also be used as a user interface.

[0033] A further aspect of the present invention relates to a method for operating a laboratory automation device as described above and below. The method can be performed by the controller.

[0034] According to embodiments of the present invention, the method comprises determining an eye position of a person from sensor data acquired by a tracking sensor and displaying information for a notification area in the work space on a transparent display such that the information is displayed to a person in front of the notification area. It is to be understood that features of the method as described above and below can be features of the laboratory automation device as described above and below.

[0035] A further aspect of the present invention relates to a computer program for a controller of a laboratory automation device, which computer program is adapted to perform a method as described above and below when executed by a processor. A further aspect of the present invention relates to a computer readable medium having such a computer program stored therein. The controller of the laboratory automation device can comprise a processor, which executes the computer program, which can be stored in a memory of the controller.

[0036] The computer readable medium can be a hard disk, a USB (Universal Serial Bus) storage device, a RAM (Random Access Memory), a ROM (Read-Only Memory), an EPROM (Erasable Programmable Read-Only Memory), or a flash memory. The computer readable medium can also be a data communication network, e.g. the Internet, which allows downloading of program code. Generally speaking, the computer readable medium can be a non-transitory or transitory medium.

[0037] A further aspect of the application relates to a controller of a laboratory automation device (10) adapted to perform a method as described herein. The method can be implemented at least partially in hardware, such as an FPGA.

[0038] These and other aspects of the application will be apparent from the reference to the examples described hereinafter and will be elucidated by the reference to those examples. BRIEF DESCRIPTION OF DRAWINGS

[0039] Embodiments of the application are described in more detail below, with reference to the accompanying drawings.

[0040] Figure 1 A schematic perspective view of a laboratory automation device according to an embodiment of the application is shown.

[0041] Figure 2 A schematic cross-sectional view of a laboratory automation device according to a further embodiment of the application is shown.

[0042] Figure 3A 、 Figure 3B and Figure 3C A schematic cross-sectional view of a laboratory automation device according to a further embodiment of the application is shown.

[0043] Figure 4 A flow chart illustrating a method for operating a laboratory automation device according to an embodiment of the application is shown.

[0044] The reference signs used in the drawings, and their meanings, are listed in abbreviated form in the list of reference signs. In principle, identical parts have the same reference signs in the figures. DETAILED DESCRIPTION

[0045] Figure 1 A laboratory automation device 10 is shown, which comprises a workbench 12 to which several components 14 of the laboratory automation device 10 are mounted. The shown example comprises a rack 14a with pipette tips 14b, a rack 14c with test tubes 16a, a microplate 14d with wells 16b, and a container 16c containing a reagent 18. In general, some of the components 14 can be liquid containers 16.

[0046] The laboratory automation device 10 further comprises a pipetting arm 20 with a pipette 22, which can be moved in three dimensions, for example by means of a motor. A sample can be removed from a test tube 16a by suction with the pipette 22 and can be dispensed into a well 16b. Similarly, a reagent 18 can be delivered into a well 16b. By means of a gripper arm 23 equipped with a gripper 24, a microplate 14d can be exchanged and moved to further devices, such as a heater, an optical analysis device, etc.

[0047] The components 14, 20, 22, 24 of the laboratory automation device 10 are arranged in a work space 26 above a workbench 12, which is enclosed by a housing 28. At the front side, the housing 28 has a door 30, which can be opened to access the components 14, 20, 22, 24. A part of the door 30 or the complete door 30 is a transparent display 32, allowing to see into the work space 26 and on which information 34 can be displayed. Such information can include symbols, text and images 34a as well as outlines and / or highlights 34b of the components 14, 20, 22, 24.

[0048] Some or all of the information 34 can be displayed with respect to a specific viewing perspective of a person in front of the laboratory automation device 10, such that the information 34 is displayed for this person near or in front of the components 14, 20, 22, 24.

[0049] In general, the information 34 can contain static information 34a, which is displayed on the same position on the transparent display 32, independent of the person’s eye position (and optionally independent of the position of the door 30). Furthermore, the information 34 can contain augmented reality information 34b, which is displayed on the transparent display 32 at a position dependent on the person’s eye position (and optionally dependent on the position of the door 30).

[0050] For example, outlines and / or highlights 34b of the components 14, 20, 22, 24 can be displayed, such that they cover the area in which the components 14, 20, 22, 24 are arranged in the work space.

[0051] To determine the eye position, the laboratory automation device 10 comprises a tracking sensor 36, which comprises a camera 36a and / or an eye tracking sensor 36b. For example, the eye tracking sensor can comprise one or more infrared light sources 36c in addition to the camera 36a. The tracking sensor 36 and in particular the camera 36a and the infrared light sources 36c can be attached to the housing 28.

[0052] Furthermore, Figure 1A controller 38 of the laboratory automation device 10 is shown, which can perform determining the eye position and rendering the information 34 to be displayed on the transparent display 32. The controller 38 can be a computer or an embedded device in data communication with the laboratory automation device 10 or can be included into the laboratory automation device 10.

[0053] Figure 2 An embodiment of the laboratory automation device 10 with a hinged door 30 is shown. Figure 1 The door 30 of the laboratory automation device 10 can be designed as follows. The door 30 can be hinged around an axis A. In a first position 40a, the door 30 is closed and the door 30 prevents access to the workspace 26 through the opening which is closed by the door 30. In a second position 40b, the door 30 is open such that a person 42 can look through the transparent display and can reach into the workspace 26, for example to exchange a liquid container 16. There can be a further position with the door 30 fully open. The position 40a, 40b of the door is determined by a position sensor 44.

[0054] The controller 38 can use the position sensor 44 to stop operation of the pipetting arm 20 and / or the gripper 24 when the door 30 is not fully closed, i.e. not in the position 40a.

[0055] Figure 2 It is also illustrated how the controller 38 determines a display area 50 of the information 34b. By the known eye position 46, a notification area 48 in the workspace is projected onto the transparent display 32. This projection serves as the display area 40 in which the information 34b is displayed. The notification area 48 can be a volume in the workspace 26. The display area 40 can be a two-dimensional shape, such as a polygonal portion of the display.

[0056] To perform the projection of the notification area 48, the controller 38 has to know the eye position 46 and the position of the transparent display 32. This position can be determined from the door position 40a, 40b or can be assumed to be constant.

[0057] Figure 3A 、 Figure 3B and Figure 3C An embodiment of the laboratory automation device 10 with a sliding door 30 is shown. Figure 1The door 30 of a laboratory automation device 10 can be designed as follows. The sliding door 30 can prevent access to the work space 26 in a first door position 40a (position "closed"), can allow access to the work space 26 in a second door position 40b (position "workbench accessible"), and can allow access to the entire work space 26 in a third door position 40c (position "fully open"). Likewise, the display area 50 can be determined depending on the door position 40a, 40b. The sliding door can have different "workbench accessible" 40b positions that vary over the height d of the access gap 52.

[0058] Figure 4 A flow chart of a method for operating a laboratory automation device 10, such as shown in the preceding figures, is shown.

[0059] In step S10, a person 42 is selected when more than one person 42 is in front of the door 30. The controller 38 can determine how many persons are in front of the door 30 by evaluating the video stream of the camera 36a. The camera 36a can be arranged such that its field of view is directed to the area in front of the door 30 where the person 42 is expected to be. The person 42 or the person's head can be identified with object recognition.

[0060] It is possible that the person 42 is selected via gesture detection by executing sensor data with the tracking sensor 36, in particular the camera 36a. For example, a waving arm or a specific gesture can be determined with object recognition of the hand and / or arm and tracking its position.

[0061] It is also possible that the person 42 is selected via the transparent display 32. The head and / or face of all persons 42 in the field of view of the camera 36a can be displayed on the transparent display 32, for example, as static information 34a. The transparent display 32 can comprise a touch screen and a person can select the person by touching the area with the corresponding static information 34a.

[0062] In step S12, the eye position 46 of the person 42 is determined from sensor data acquired by the tracking sensor 36. For example, the eye position 46 can be extracted from an image of the head of the selected person in step S10 or from a single person in front of the door 30. The eye position 46 can be determined via object recognition in the video stream from the camera 36a. It is also possible that an infrared light source 36c is used, infrared light is reflected from the eyes and those reflections can be identified in the video stream.

[0063] The eye position 46 can be a three-dimensional coordinate provided with respect to a coordinate system in which also the positions of points inside the work space 26 can be calculated and / or in which also the position of the transparent display 32 is known.

[0064] In step S14, the augmented reality information 34b for the notification area 48 in the work space 26 is displayed on the transparent display 32.

[0065] The controller 38 determines one or more notification areas 48 for which to display information 34b. The notification area 48 can be an area in which a component 14 of the laboratory automation device 10 is located. The information 34b can be status information of the component 14, such as an error message or a message to a person to replace or refill the component 14. It can also be that the notification area 48 is a blank area in the work space 26. For example, an area in which a component 14 of the laboratory automation device 10 is to be placed by the person 42. The information 34b can be status information of the component 14 to be placed.

[0066] The controller 38 can maintain an internal model of the work space 26 and the components 14 inside. The model can provide information on which components 14 are located in which positions. The model can also provide information on boundary volumes of the components. Such boundary volumes can be used as notification areas 48. The internal model can be updated when components 14 are placed into the work space 26 and removed from the work space 26. The internal model can be updated when components 14 change their position, for example due to operation of the pipetting arm 20 and / or the gripper 24.

[0067] When one or more notification areas 48 are determined, the notification areas 48 are projected onto the transparent display 32 towards the eye position 46 to determine a display area 50 on the transparent display 32. The transparent display 32 can be modeled as a rectangle whose position is determined by the positions 40a, 40b, 40c of the door 30. The notification areas 48 can be projected onto this rectangle via a central perspective with respect to the eye position 46.

[0068] Finally, the information 34b shown in the corresponding display area 50 is rendered. The information 34b can comprise at least one of text, numbers, symbols, outlines, highlighting, etc. It can be that the graphical information 34b flashes or changes color to catch the attention of the person 42.

[0069] Due to the projection, the information 34b is displayed from the perspective of the person 42 in front of the notification area 48. In this way, the person can easily identify to which component 14 the information 34b belongs, in particular independent of the position 40a, 40b, 40c of the sliding door 30 Figures 3A to 3C or the opening angle of the hinged door 30 Figure 2 .

[0070] While the application has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the application is not limited to the disclosed embodiments. Other variations of the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed application, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit can fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.

[0071] List of reference signs

[0072] 10 laboratory automation device

[0073] 12 worktable

[0074] 14 assembly

[0075] 14a rack

[0076] 14b pipette tip

[0077] 14c rack

[0078] 14d microplate

[0079] 16 liquid container

[0080] 16a test tube

[0081] 16b well

[0082] 16c container

[0083] 18 reagent

[0084] 20 pipetting arm

[0085] 22 pipette

[0086] 23 gripper arm

[0087] 24 gripper

[0088] 26 work space

[0089] 28 housing

[0090] 30 door

[0091] 32 transparent display

[0092] 34 information

[0093] 34a static information

[0094] 34b augmented reality information

[0095] 36 tracking sensor

[0096] 36a camera

[0097] 36b eye tracking sensor

[0098] 36c infrared light source

[0099] 38 controller

[0100] 40a first door position

[0101] 40b second door position

[0102] 40c third door position

[0103] 42 person

[0104] 44 door position sensor

[0105] 46 eye position

[0106] 48 notification area

[0107] 50 display area

[0108] 52 reach gap

Claims

1. A laboratory automation device (10), comprising: A workspace (26) and a pipette arm (20), wherein the workspace (26) has a liquid container (16) and the pipette arm (20) is used to move liquid between the liquid containers (16); A housing (28) that encloses the working space (26); The door (30) of the housing (28) is for accessing the workspace (26), wherein the door (30) includes a transparent display (32) for displaying information (34) and allowing a person to see inside the workspace (26); A tracking sensor (36) is used to track the position (46) of a person's (42) eyes; The laboratory automation equipment (10) is suitable for: The position (46) of the person's (42) eyes is determined based on sensor data acquired by the tracking sensor (36); Information (34b) for a notification area (48) in the workspace (26) is displayed on the transparent display (32) such that the information (34b) is displayed from the perspective of a person (42) in front of the notification area (48); The notification area (48) is projected onto the transparent display (32) facing the eye position (46) to determine the display area (50) on the transparent display (32), and the information (34b) is displayed in the display area (50).

2. The laboratory automation equipment according to claim 1, in, The notification area (48) is the area in which the components (14) of the laboratory automation equipment are located; The information (34b) displayed on the transparent display (32) is the status information of the component (14).

3. The laboratory automation equipment (10) according to claim 1 or 2, in, The notification area (48) is the area in which the components (14) of the laboratory automation equipment (10) will be placed; The information (34b) displayed on the transparent display (32) is the status information of the component (14) to be placed.

4. The laboratory automation equipment (10) according to claim 1 or 2, in, The tracking sensor (36) includes a camera (36a).

5. The laboratory automation equipment (10) according to claim 1 or 2, in, The tracking sensor (36) includes an eye-tracking sensor (36b).

6. The laboratory automation equipment (10) according to claim 1 or 2, in, The tracking sensor (36) is adapted to determine more than one person (42) in front of the door (30).

7. The laboratory automation equipment (10) according to claim 6, in, The person (42) whose eye position is to be determined is selected by gesture detection performed through sensor data from the tracking sensor.

8. The laboratory automation equipment (10) according to claim 6, in, The person (42) whose eye position (46) is to be determined is selected via the transparent display (32).

9. The laboratory automation equipment (10) according to claim 1 or 2, in, The door (30) has a position sensor (44) for determining the position (40a, 40b, 40c) of the door; The display area (50) on the transparent display (32) is determined based on the position (40a, 40b, 40c) of the door (30), wherein the information (34b) is displayed in the display area (50).

10. The laboratory automation equipment (10) according to claim 1 or 2, in, The transparent display (32) includes a touchscreen.

11. A method for operating a laboratory automation device (10) according to any one of claims 1-10, the method comprising: The position (46) of the person's (42) eyes is determined based on sensor data acquired by the tracking sensor (36); Information (34b) for a notification area (48) in the workspace (26) is displayed on the transparent display (32) such that the information (34b) is displayed from the perspective of a person (42) in front of the notification area (48).

12. A computer program product for a controller (38) of a laboratory automation device (10), comprising a computer program adapted to perform the method according to claim 11 when executed by a processor.

13. A computer-readable medium storing a computer program in a computer program product according to claim 12.

14. A controller (38) for a laboratory automation device (10), said controller (38) being adapted to perform the method according to claim 11.

Images