Pipetting detection device and method for detecting at least a three-dimensional position of a pipette tip

Abstract

The invention relates to a pipetting detection device (1) with a plurality of reference markings (5) that can be arranged or are arranged in a fixed position relative to a multiwell plate (3), each reference marking (5) of the plurality of reference markings (5) having at least one reference point (6), an optical sensor (7) that can be arranged or is arranged and configured in a fixed position relative to a pipette (9) such that at least a subset of the reference points (6) can be optically detected by means of the optical sensor (7), and a control device (13) that is operatively connected to the optical sensor (7) and configured to determine a three-dimensional position of the pipette tip (11) on the basis of the detected reference points (6).

Description

[0001] The invention relates to a pipetting detection device and a method for detecting an at least three-dimensional position of a pipette tip.

[0002] DE 10 2020 208 740 A1 discloses a method for determining an absolute coordinate for a well of a multiwell plate using an optical sensor attached to a pipette, as well as a suitable addressing device. However, this method only allows the determination of a two-dimensional position in a coordinate system spanned by a surface or frame of the multiwell plate, making it difficult, only with comparatively low accuracy or robustness, or even impossible—especially with larger multiwell plates—to determine the exact well into which pipetting is actually taking place. Furthermore, no additional information regarding the immersion depth of the pipette, the pipette trajectory, or the exact position of the tip end of a pipette tip in use can be obtained.In some cases, systems are used to record and / or monitor pipetting processes that use spatially fixed optical sensors that are separate from the pipette. However, these are difficult to calibrate and extremely susceptible to small deviations in the relative position between the optical sensor and the multi-well plate with reference to a predetermined reference position. Furthermore, highly accurate, three-dimensional recording of the pipetting process is hardly possible due to the comparatively large distance between the optical sensor and the individual wells.

[0003] The invention is therefore based on the object of creating a pipetting detection device and a method for detecting an at least three-dimensional position of a pipette tip, wherein the aforementioned disadvantages are at least reduced, preferably do not occur.

[0004] The object is achieved by providing the present technical teaching, in particular the teaching of the independent claims as well as the embodiments disclosed in the dependent claims and the description.

[0005] The object is achieved in particular by providing a pipetting detection device having a plurality of reference markings that can be arranged or are arranged in a predetermined marking position relative to a multi-well plate, each reference marking of the plurality of reference markings having at least one reference point. The pipetting detection device also has an optical sensor that can be arranged or is arranged and configured in a predetermined position relative to a pipette - in particular in a predetermined sensor position - such that at least a minority of the reference points can be optically detected by means of the optical sensor. In addition, the pipetting detection device has a control device that is operatively connected to the optical sensor and configured to determine a three-dimensional position of the pipette tip based on the detected reference points.Advantageously, the optical sensor, which can be arranged or positioned in a fixed position relative to the pipette – and thus in particular is carried along with it – in combination with the reference markings, which can be arranged or positioned in a fixed position relative to the multiwell plate, enables highly precise three-dimensional recording of a pipetting process, in particular with high precision depth information perpendicular to a surface of the multiwell plate facing the pipette. In particular, submillimeter accuracy can be achieved in the depth direction, i.e., in the immersion direction of the pipette tip into the wells, referred to without loss of generality as the z-direction. This, in turn, advantageously enables precise digitization of even manual work steps, whereby, in particular, automatic logging for work documentation is possible. Determination of the position of the pipette tip in real time is particularly advantageous.

[0006] A three-dimensional position is not understood as exhaustive in the context of the present technical teaching. Such a position encompasses at least three dimensions, but it can encompass more than three dimensions, in particular six dimensions.

[0007] In the context of the present technical teaching, a reference marking is understood as a combination of a unique, in particular one-to-one, identifier and at least one reference point of the plurality of reference points. A reference point is understood as a geometric point or location whose coordinates in a predetermined coordinate system—in particular in a multiwell plate coordinate system fixed on the multiwell plate or defined by the multiwell plate, or in a coordinate system whose position relative to the multiwell plate coordinate system is known—are either known themselves or can be calculated based on known coordinates.An identifier is understood to be a designation assigned to the reference point, for example a numeric or alphanumeric designation or a name, a color or a geometric shape, or generally any feature that is suitable for distinguishing it from other reference points. The identifier can be arranged in the area of ​​the reference point in a way that is recognizable to the optical sensor, or it can be stored virtually in such a way that it can be assigned to the reference point in an image taken by the optical sensor. In particular, the coordinates of the reference point are linked to the identifier, so that the coordinates can be deduced or accessed using the identifier. It is possible for exactly one and only one reference point to be assigned to a reference marking; in particular, a suitably marked or otherwise customizable reference point can be identical to the reference marking assigned to it.However, it is also possible for a reference marking, and thus in particular an identifier, to be assigned a plurality of reference points, for example, three reference points or four reference points, whose coordinates can be determined using the identifier, in particular because they are either known or because they can be calculated from a known coordinate of the reference marking, for example, a center point or center of gravity. For example, a reference marking configured as a square two-dimensional code—in particular with a known size—can have its corner points as four reference points.

[0008] In particular, a reference marker comprises exactly one identifier and at least one reference point.

[0009] In an embodiment in which a reference marking has at least three reference points, preferably four reference points, it is sufficient for determining the position of the pipette tip if the optical sensor can detect exactly one reference marking. However, the accuracy of the determination can be increased—in particular, a determination error can be reduced—if the optical sensor can detect more than one reference marking simultaneously.

[0010] From a mathematical perspective—without wishing to be bound by theory—it is sufficient to determine the position of the pipette tip if three reference points are recorded simultaneously. However, depending on the specific algorithm used to determine the position, it may be necessary or advantageous to record more than three reference points simultaneously.

[0011] In particular, there are algorithms that require the simultaneous acquisition of at least four reference points or even more than four. In any case, however, the accuracy of position determination increases with the number of acquired reference points and, in particular, the number of acquired reference markers.

[0012] In particular, the distances between the reference points or the length ratios of the reference markings to a known geometry are important for determining the position. If the detected reference points are as far apart as possible in space, the position can be determined with high accuracy. However, if the detected reference points are close to each other, the spatial direction is extrapolated and therefore quickly becomes inaccurate, for example due to component tolerances, subpixel deviations in the image, and other possible effects. These inaccuracies are independent of the spatial direction; therefore, if a series of reference points is detected in the x-direction without loss of generality, the accuracy in the y-direction is not improved and is based only on the distances between the detected reference points in the y-direction.It is therefore always desirable that the distances between the detected reference points or reference markings in the x- and y-direction are as large as possible, or are sufficiently high in relation to the geometry to be recognized.

[0013] In particular, the at least three-dimensional position of the pipette tip can be determined based on a perspective representation, in particular a distortion of the reference points, in particular a distortion of at least one reference marking, in the optical recording of the optical sensor.

[0014] In one embodiment, the control device is configured to determine a six-dimensional position of the pipette tip based on the detected reference points. In particular, the six-dimensional position of the pipette tip can be determined based on a perspective representation, in particular distortion of the reference points, in particular at least one reference marking, in the optical image of the optical sensor.

[0015] In the context of the present technical teaching, determining the position of the pipette tip means, in particular, that this position is determined in the multiwell plate coordinate system. Furthermore, this means, in particular, that the position of the optical sensor is determined in the multiwell plate coordinate system. If the position of the pipette tip in a sensor coordinate system of the optical sensor is known, for example, through one-time or multiple calibrations, continuous acquisition, or in another way, the position of the pipette tip in the multiwell plate coordinate system can always be deduced from the determined position of the optical sensor in the multiwell plate coordinate system.

[0016] In one embodiment, the predetermined marking positions are known. Alternatively or additionally, the predetermined sensor position is known.

[0017] In particular, the reference markings can be arranged or arranged in a fixed position relative to the multiwell plate.

[0018] In the context of the present technical teaching, "stationary" describes in particular a stable, three-dimensional position, in particular a center of gravity or center point position—especially in Cartesian coordinates. In contrast, "positionally fixed" in the context of the present technical teaching describes in particular a stable, six-dimensional position or orientation, including the three rotation angles of an extended object.

[0019] In one embodiment, the reference markings are not arranged directly on the multiwell plate, but on an additional object or component, which in turn can be arranged in a fixed location, in particular in a fixed position, relative to the multiwell plate, in particular on the multiwell plate. In this case, it is possible for the multiwell plate itself not to be part of the pipetting detection device; however, in one embodiment, the multiwell plate can also be part of the pipetting detection device. In another embodiment, the reference markings are arranged directly on the multiwell plate, for example, printed, embossed into the multiwell plate, formed as a raised surface structure of the multiwell plate, glued on, or attached in some other way. In this case, the multiwell plate is part of the pipetting detection device, i.e., the pipetting detection device comprises the multiwell plate.In a particularly simple embodiment, predetermined, characteristic points of a multi-well plate can also be used as the reference points of the reference markings.

[0020] Furthermore, it is possible to use fixed, predetermined markings in a spatial environment of a multiwell plate as reference markings or reference points. The multiwell plate is then arranged in a predetermined position—and in particular, location—in the environment. For example, the multiwell plate can be arranged in a recess—with minimal play at most—by being inserted into the recess from above, for example, geodetically. The recess can also be formed or arranged, for example, on or in a laboratory table.

[0021] In one embodiment, the reference markings can be arranged or are arranged—in particular over their entire circumference—along a preferably closed circumferential line of the multiwell plate, in particular in an edge region or on an edge of the multiwell plate. Alternatively or additionally, the reference markings are arranged on or in the region of a surface of the multiwell plate toward which the wells of the multiwell plate are open, i.e., this surface faces the pipette during a pipetting process. In particular, this is a surface of the multiwell plate in which the upper edges of the wells lie. This surface is referred to below as the pipetting surface.

[0022] The distribution of the reference markings along a particularly closed circumferential line advantageously allows a sufficient minority of the reference points to be detected whenever the optical sensor is aligned with the multiwell plate, regardless of a specific current position of the optical sensor. Preferably, the reference markings can be arranged or are arranged in such a way that at least three reference points can always be detected, at least from those positions that the optical sensor assumes relative to the multiwell plate during normal pipetting processes.

[0023] In particular, at least the minority of the reference points can be optically detected by means of the optical sensor if the pipette with the optical sensor is arranged above the multiwell plate.

[0024] In the context of the present technical teaching, a minority of reference points is understood to mean, in particular, a number of reference points that is smaller than the total number of reference points. In particular, within the context of the present technical teaching, it is not necessary to detect all reference points or reference markings simultaneously using the optical sensor. Without wishing to be bound by theory, simultaneous detection of three reference points is sufficient for a sufficiently accurate determination of the position of the pipette relative to the multiwell plate, whereby each additionally detected reference point can be used to increase accuracy.

[0025] The optical sensor can be arranged or arranged in a fixed position relative to the pipette.

[0026] In particular, the optical sensor can be arranged or arranged in such a rigid, stationary manner, preferably in a fixed position, relative to the pipette that a viewing direction of the optical sensor with respect to the pipette is constant over time.

[0027] In one embodiment, the optical sensor is formed in multiple parts with the pipette and can be attached or is attached to the pipette. In this case, the pipette itself may not be part of the pipetting detection device; however, in one embodiment, the pipette may also be part of the pipetting detection device. Alternatively, the optical sensor is formed in one piece with the pipette or integrated into the pipette. In this case, the pipette is part of the pipetting detection device, i.e., the pipette is part of the pipetting detection device, i.e., the pipette is part of the pipetting detection device.

[0028] In a simple embodiment, the pipetting detection device thus comprises only the reference markings, preferably on a reference frame, the optical sensor, and the control device. In particular, in this simple embodiment, the pipetting detection device does not comprise the multiwell plate or the pipette. In another embodiment, the pipetting detection device additionally comprises the multiwell plate, in particular with the reference markings arranged or formed directly thereon. Alternatively or additionally, in yet another embodiment, the pipetting detection device comprises the pipette, in particular with the optical sensor arranged thereon or integrated into the pipette.

[0029] In one embodiment, the pipette is designed as a handheld pipette. This advantageously allows even manual pipetting processes to be automatically recorded and, in particular, logged.

[0030] In one embodiment, the control device is integrated into the pipette or arranged on the pipette. Alternatively, the control device can be configured externally to the pipette, for example, as a stationary computing device.

[0031] In one embodiment, the optical sensor is designed as a camera.

[0032] The control device is preferably operatively connected to the optical sensor - or optionally to additional sensors of the pipette - in particular via a cable connection or wirelessly.

[0033] To determine the position of the pipette tip, it is not absolutely necessary for the pipette tip itself to be imaged in the optical image of the optical sensor; rather, it is generally sufficient if the location or position of the pipette tip relative to the optical sensor, in particular relative to a pipette body, is known. Since the position of the optical sensor can be determined by detecting the reference points, the position of the pipette tip is also known at the same time.

[0034] In one embodiment, however, the optical sensor can be arranged or is arranged and configured in such a way that a pipette tip of the pipette and, at the same time, at least a minority of the reference points can be optically detected by means of the optical sensor. This advantageously allows a one-time or multiple calibration or verification of the position of the pipette tip, or even a determination of the position of the pipette tip relative to the optical sensor; alternatively or additionally, in a more complex embodiment, it is also possible to continuously track the position of the pipette tip relative to the optical sensor, for example, to determine it in each image or in every nth image of the optical sensor, where n is a natural number > 1.

[0035] In one embodiment, the optical sensor is aligned such that the pipette tip, in particular the tip end of the pipette tip, is always imaged in the receptacles of the optical sensor - ie in each receptacle.

[0036] In particular, the pipette tip and at least the minority of the reference points can be optically detected by means of the optical sensor if the pipette with the optical sensor is arranged above the multiwell plate.

[0037] According to a further development of the invention, the reference markings are arranged on a reference frame. In this way, the reference markings can advantageously be arranged in a defined edge region of the multiwell plate.

[0038] In one embodiment, the reference frame is designed to be arranged in a fixed location, in particular in a fixed position, on the multiwell plate. Advantageously, it is thus possible to use the reference markings with otherwise conventional, commercially available multiwell plates; thus, there is no need to provide multiwell plates specifically provided with the reference markings. In one embodiment, the reference frame can be attached to the multiwell plate. In another embodiment, the reference frame can be placed or slipped over the multiwell plate in such a way that it is connected to it in a fixed location, in particular in a fixed position, in particular in a fixed position—in particular by means of a form-fitting connection. Alternatively, the multiwell plate can be inserted into the reference frame—in particular from above, geodetically.

[0039] In one embodiment, it is possible for the reference frame, on the one hand, and the multiwell plate, on the other hand, to have corresponding orientation elements that are configured to interact to ensure a defined and preferably stable relative position of the reference frame to the multiwell plate. For example, the multiwell plate can have a recess, and the reference frame can have a projection or nose corresponding to and engaging with the recess—or vice versa. It is also possible for the multiwell plate and the reference frame to each have at least one beveled corner to ensure a clear relative position.

[0040] In another embodiment, the reference frame is formed integrally with the multiwell plate or is part of the multiwell plate; in particular, it is possible for the reference frame to be an edge of the multiwell plate.

[0041] The reference markings can be printed on the reference frame, embossed into the reference frame, formed raised on the reference frame, formed as a surface structure of the reference frame, or otherwise stably connected to it, in particular fastened to it, for example glued onto the reference frame.

[0042] According to a further development of the invention, the pipetting detection device also comprises the pipette, with the optical sensor being arranged on the pipette base body. This represents a particularly compact and at the same time handy design of the pipetting detection device.

[0043] In one embodiment, the optical sensor is attached to the pipette base body. Alternatively, the optical sensor is integrated into the pipette base body. Alternatively or additionally, the optical sensor is formed as a single piece with the pipette base body.

[0044] According to a further development of the invention, the pipette base body comprises a pipette tip receptacle on which the pipette tip can be exchangeably arranged. Advantageously, this makes it easy to exchange the pipette tip, particularly for pipetting different substances and / or different substance volumes.

[0045] In one embodiment, the pipette is designed as a graduated micropipette, for example, an Eppendorf pipette. Alternatively or additionally, the pipette is designed as a single-channel pipette or a multi-channel pipette. Within the scope of the present technical teaching, it is readily possible to detect more than one pipette tip with the optical sensor and to determine their respective positions in at least three dimensions.

[0046] According to a further development of the invention, the pipetting detection device comprises a display device that is operatively connected to the control device and configured to display at least one piece of information relevant to a pipetting process to be performed. In this way, an operator operating the pipette can advantageously be supported, instructed, or guided by means of information displayed by the display device, making the pipetting process easier and less prone to error. In particular, the display device can be used to show the operator a well that is to be addressed next or to highlight it—for example, in the manner of augmented reality (AR)—into or from which pipetting is to take place next.

[0047] In the context of the present technical teaching, the term "display" is not limited to a visual display, although a visual display represents one possible embodiment. However, the display device can additionally or alternatively be configured to display or output the relevant information in another way, for example acoustically, haptically, or in the form of a vibration. For example, the information can be output in Braille, or the display device can be configured to vibrate the pipette when the pipette tip is positioned over a well to be addressed.

[0048] In the context of the present technical teaching, a well to be addressed is generally understood to be a well into or from which pipetting is to take place.

[0049] In particular, the display device—in combination with the control device—is configured to display the information relevant to the pipetting process to be performed depending on the determined—at least three-dimensional—position of the pipette tip. Advantageously, the information can thus be displayed, particularly depending on the progress of the pipetting process, and thus always in a current state. Furthermore, it is possible to mark and highlight relevant wells in the current vicinity of the pipette tip on or by means of the display device.

[0050] In one embodiment, the display device - in combination with the control device - is configured to display or output coordinates of a well to be addressed next.

[0051] In one embodiment, the display device is designed as a screen separate from the pipette and the multiwell plate. The display device can also be designed as a device, e.g., a headset or glasses, to be worn by the operator, in particular as a body-worn head-up display. Furthermore, the display device can also be designed as a display device attached to the pipette, formed integrally with the pipette, or integrated into the pipette, in particular as a monitor or as a head-up display carried with the pipette.

[0052] In yet another embodiment, the display device can have a plurality of separate light sources, preferably LEDs, arranged, for example, in the form of a coordinate system or in the form of a two-dimensional matrix, by means of which information can be displayed, in particular individual wells of the multi-well plate can be highlighted or illuminated. The light sources can be single-colored or multi-colored. In one embodiment, each well of the multi-well plate can be uniquely, in particular uniquely, assigned a light source from the plurality of light sources. In particular, such a display device can be configured to be arranged beneath the then preferably transparent or translucent multi-well plate, so that the light sources are visible through the multi-well plate and in particular through the individual wells.This allows individual wells to be highlighted in a particularly simple and intuitive way by controlling the correspondingly assigned light source.

[0053] In yet another embodiment, the display device may comprise a projection device or be configured as a projection device, wherein the projection device is configured to project information, in particular an image or symbol, onto the multiwell plate. In this way, too, individual wells, in particular, can advantageously be marked or highlighted by means of such a projection.

[0054] According to a further development of the invention, the reference markings are each designed as a two-dimensional code. This advantageously represents a design of the reference markings that can be detected particularly well and, in particular, with high accuracy by the optical sensor. In particular, such a two-dimensional code can be oriented such that the representation of the code in the two-dimensional image plane of the sensor, in particular the distortion of the code, allows conclusions to be drawn about the pose or orientation of the optical sensor relative to the code.

[0055] In another, simpler embodiment, the reference markings can also be simple characters, unique letters and / or numbers, as well as combinations thereof, pictograms, or patterns. It is important that each reference marking is clearly, and in particular unambiguously, assigned a sign that is distinct and distinguishable from the other signs.

[0056] The at least one reference point that a reference marking has is preferably a predetermined, distinguished point of the reference marking, for example a center point, center of gravity, end point, or corner point. Its coordinates in the multi-well plate coordinate system are uniquely assigned to the reference marking, in particular uniquely assigned, or can be calculated based on at least one known coordinate of the reference marking. For example, it is possible for the coordinates of a center point to be known for a reference marking, wherein the coordinates of a corner point of the same reference marking can be calculated based on a known size of the reference marking and the coordinates of the center point. Alternatively, the coordinates of at least one corner point of the reference marking can also be known.

[0057] In one embodiment, the two-dimensional code is selected from a group consisting of AprilTag, ArUco-Tag, ARTag, ARToolKit, Intersense, RUNE-43, RUNE-129, reacTIVIsion, Framedge, and QR code. These codes are particularly suitable for use as reference markers in the context of the present technical teaching. This applies particularly to AprilTag, since such codes can be detected with particularly high levels of security and accuracy. Furthermore, due to the comparatively low computational effort, real-time position determination is also possible on a microcontroller. In principle, however, other two-dimensional codes are also possible as reference markers. In general, the reference markers are preferably designed as so-called fiducial markers or designed in such a way that they can be used as fiducial markers.

[0058] In particular, if the two-dimensional code is square, for example in the case of AprilTag, such a reference mark can have its vertices as four reference points.

[0059] In the simplest embodiment, the reference markings each carry, as information, a unique, in particular one-to-one, designation of the respective reference marking as the identifier. Together with the predetermined, in particular known, marking position at which the respective unique designation in the form of the reference marking is arranged, the position of the optical sensor relative to the respective reference marking and thus also relative to the multiwell plate coordinate system, in which the marking position is known, can be determined by evaluating the corresponding signal from the optical sensor, in particular by image analysis.The unique designation encoded in the respective reference marking or stored in another way is therefore clearly, in particular one-to-one, assigned a known marking position in the multi-well plate coordinate system, preferably at the same time a known position or pose of the reference marking, wherein the representation of the reference marking in the image plane of the optical sensor can then be used to infer the (3D) position and / or (6D) position of the optical sensor relative to the reference marking and thus also in the multi-well plate coordinate system. In one case, it can be assumed that the multi-well plate is arranged in a spatially fixed manner; the multi-well plate coordinate system is then also a spatially fixed, i.e. in particular a laboratory-fixed coordinate system. However, it is also possible for the multi-well plate and thus also the multi-well plate coordinate system to move in space - preferably uniformly.Accelerated movement is also possible in principle. If three reference points are detected by the optical sensor simultaneously, its three-dimensional position in the multi-well plate coordinate system can be determined; at the same time, the three-dimensional position of the pipette tip in the multi-well plate coordinate system can also be determined, since the optical sensor is arranged at the predetermined and, in particular, known sensor position relative to the pipette. As explained above, this is even the case if the pipette tip itself is not detected by the optical sensor, if the position of the pipette tip relative to the optical sensor is known with sufficient accuracy. However, the accuracy of the position determination is increased if the pipette tip is also detected by the optical sensor. Furthermore, the accuracy of the position determination increases with each additional detected reference point.In particular, the additionally recorded reference points can be used for error compensation.

[0060] In particular, accuracy is increased, at least when certain algorithms are used, when reference points are recorded that are as far apart as possible in the image, as this allows the geometric relationships to be resolved more precisely. This applies, without loss of generality, independently for the x and y directions, i.e. good accuracy in the x direction does not automatically mean good accuracy in the y direction if the 3D geometry is derived from them. This is why it is particularly advantageous to record multiple reference points, in particular multiple reference markings. In particular, it is possible in an algorithm to sort the reference markings and only those that are further apart from each other, in particular those furthest apart in the image, are used for evaluation for each coordinate.Alternatively, it is also possible in another algorithm not to sort out the intermediate reference markings, but to use them, in particular in a statistical procedure, to improve the accuracy of the position determination.

[0061] In one embodiment, the pipetting detection device comprises at least one inertial sensor, also referred to as an inertial sensor (Ineratial Measurement Unit - IMU), which can be arranged or is arranged on the pipette, in particular integrated into the pipette, and which is operatively connected to the control device. With the aid of the at least one inertial sensor, the movement of the pipette can advantageously be detected even more precisely. In particular, a displacement of the pipette can be further tracked and / or extrapolated even in a situation in which fewer than three reference points or no reference point can be detected—for example, due to motion blur or because the reference points are currently outside the field of view of the optical sensor.It is also possible to increase the sampling rate using the signals from at least one inertial sensor, since such inertial sensors can typically be sampled at higher sampling rates than the optical sensor. Without wishing to be bound by theory, such an inertial sensor provides linear acceleration values ​​in three dimensions as well as three independent angular velocities. By integrating twice, a relative position can be calculated from the acceleration values, which can be added to a known or assumed starting position to calculate a current actual position.In this way, the at least one inertial sensor in combination with the optical sensor enables very precise position tracking, in particular by fusing the slower but precise absolute optical position determination with the faster scanned relative position determination by the at least one inertial sensor, particularly under the condition that only the pipette, but not the multiwell plate, moves in a given gravitational field, or that the pipette and the multiwell plate are arranged at least in a common inertial system. Alternatively or additionally, the motion detection by the at least one inertial sensor allows image enhancement by reducing motion blur, particularly in a deconvolution process.

[0062] The pipetting detection device may comprise additional sensors, in particular to increase the accuracy of the position determination.

[0063] According to a further development of the invention, it is provided that the control device is configured to carry out a method according to the invention described below or a method according to one or more of the embodiments described below.

[0064] The object is also achieved by providing a method for detecting a - in particular at least - three-dimensional position of a pipette tip of a pipette relative to a multi-well plate, which method comprises the following steps: optically detecting a plurality of reference points arranged in predetermined positions relative to the multi-well plate by an optical sensor arranged stationary on the pipette, wherein the reference points are assigned to a plurality of reference markings, wherein each reference marking of the plurality of reference markings has at least one reference point of the plurality of reference points, and determining the three-dimensional position of the pipette tip based on the detected reference points. In connection with the method, in particular those advantages arise which were already explained above in connection with the pipetting detection device.

[0065] Preferably, the method uses a pipetting detection device according to the invention or a pipetting detection device according to one or more of the previously described embodiments. Alternatively or additionally, the method preferably includes at least one method step, in particular a combination of method steps, which has / have been previously described explicitly or implicitly in connection with the pipetting detection device.

[0066] In particular, a minority of a larger majority of reference points or reference markings are simultaneously detected by the optical sensor.

[0067] In one embodiment, the pipette is hand-held, i.e. moved manually.

[0068] According to a further development of the invention, the pipette tip is additionally detected by the optical sensor. This can be done once or repeatedly for calibration purposes, or continuously, particularly in each image captured by the optical sensor.

[0069] According to a further development of the invention, a two-dimensional code is used as the reference markers, wherein the two-dimensional code is selected in particular from a group consisting of AprilTag, ArUco-Tag, ARTag, ARToolKit, Intersense, RUNE-43, RUNE-129, reacTIVIsion, Framedge, and QR code. In a preferred embodiment, the code is AprilTag. Other fiducial markers can also be used as reference markers.

[0070] According to a further development of the invention, a perspective projection, in particular based on the model of a pinhole camera, is used to determine the position of the pipette tip. This proves particularly advantageous due to the spatial proximity of the optical sensor to the imaged wells, precisely because greater distortions occur compared to a sensor arrangement located further away.

[0071] In one embodiment, the position of the pipette tip is determined using homogeneous coordinates, in particular augmented by a scaling coordinate. In particular, using a projection matrix that describes the internal and external orientation of the optical sensor, points in three-dimensional space—in particular the reference points—are projected onto a two-dimensional plane, in particular the image plane of the optical sensor. Conversely, the position and orientation of the recording optical sensor can be determined from known points in three-dimensional space—specifically the reference points or reference markings at the marking positions—and their associated coordinates in the two-dimensional image, if its calibration matrix is ​​known and a minimum number of known pixels, in particular three pixels, are present.

[0072] In one embodiment, a perspective-n-point (PnP) problem is solved based on the acquired reference points. This solution then provides, in particular, the position and attitude or pose of the optical sensor relative to the multiwell plate coordinate system.

[0073] According to a further development of the invention, a distance between the pipette tip and the optical sensor is determined—in particular calibrated—by placing the pipette tip in a position determined by at least one reference marking of the plurality of reference markings, wherein the distance is determined based on the at least one reference marking. In particular, the distance of the tip end of the pipette tip to the image plane of the optical sensor is calibrated. In particular, by calibrating the distance between the pipette tip and the optical sensor, submillimeter accuracy can advantageously be ensured. In order to avoid damaging the tip end, the pipette tip is preferably placed with slight force for distance determination.

[0074] The distance is understood here in particular to be a distance along a line of sight that is perpendicular to the image plane of the optical sensor and in particular passes through a center point of the image plane. Alternatively or additionally, the distance is understood, without loss of generality, to be a distance in the z-direction of the sensor coordinate system of the optical sensor, or in the direction of a longitudinal axis of the pipette, in particular of the pipette base body, also referred to as the pipette axis for short.

[0075] To determine or calibrate the distance, it is sufficient to detect exactly one reference mark, provided this reference mark has at least three reference points, preferably four. Otherwise, the distance can also be determined or calibrated relative to a position determined by more than one reference mark.

[0076] If the size of the reference mark in the multiwell plate coordinate system is known, the distance can be determined solely from the size depicted in the image of the reference mark, i.e. in the image plane of the optical sensor.

[0077] In one embodiment, the distance can be determined or calibrated once, whereby it can be assumed that even when the pipette tip is replaced, at least no relevant deviations in the distance between the tip end and the optical sensor typically occur. However, the distance can also be determined or calibrated regularly or as needed, especially if a deviation is assumed due to, for example, a bend in the pipette tip or other reasons.

[0078] Here and in the following, the "tip end" refers in particular to the outermost tip end, i.e., the end of the pipette tip located most distally from the operator's perspective. Thus, the tip end is in particular the end of the pipette tip facing away from the pipette body.

[0079] Alternatively or additionally, the image position of the pipette tip, in particular the tip end of the pipette tip, in the image plane of the optical sensor is determined once, repeatedly, in particular regularly or as needed, or continuously in each image captured by the optical sensor. An image captured by the optical sensor is also synonymously referred to as an image or captured image.

[0080] In particular, the distance is determined based on the at least one reference mark and the image position of the pipette tip, in particular the tip end of the pipette tip. The image is recorded when the pipette tip is placed in the position determined by the at least one reference mark.

[0081] From the predetermined, known sensor position, the position of the pipette axis is known in the sensor coordinate system comprising the image plane of the optical sensor as the origin plane, in particular, without loss of generality, the x,y plane at z = 0. By detecting the at least one reference marking, preferably at least three reference points, their position in both the sensor coordinate system and the multi-well plate coordinate system is also known, and thus at the same time the distance between the image plane of the optical sensor and the tip end is known, since the tip end is placed in the known position determined by the at least one reference marking. The angle of a line of sight between the optical sensor and the tip end is obtained from the image position of the tip end in the recorded image.From the information thus obtained, a complete position of the pipette tip in space with six degrees of freedom can now be easily determined.

[0082] Based on this calibration, the position of the pipette axis in space can be determined at any time during the pipetting process. In particular, impermissible angular alignment during pipetting, such as an excessively shallow angle, can be detected and logged. Alternatively or additionally, a warning can be issued if the pipette axis is held at an impermissible angle, for example, visually, acoustically, haptically, or through vibration of the pipette.

[0083] As already explained above, continuous detection of the pipette tip by the optical sensor or continuous evaluation of the image position of the pipette tip is not necessary if the position of the pipette tip, in particular the tip end, relative to the optical sensor has been determined once and is thus known. It can typically be assumed that the position of the tip end remains constant with sufficient accuracy even when the pipette tip is replaced. However, as also explained above, it is certainly possible to repeatedly determine the position of the pipette tip, in particular the image position of the tip end, and in particular to continuously track it.

[0084] According to a further development of the invention, an image position of the outermost tip end of the pipette tip is determined in a recording of the optical sensor by detecting edge lines of the pipette tip in the recording and determining the image position of the tip end based on the detected edge lines. This enables reliable determination of the image position of the tip end, in particular by means of an image processing algorithm. Advantageously, it can also be determined whether a pipette tip is arranged on the pipette base body at all. The angle that the two edge lines enclose with each other is distorted by the perspective but can be estimated in advance. The further the optical sensor is arranged perpendicular to the pipette axis, the more acute or smaller the angle enclosed in the image by the edge lines.

[0085] The image position or - equivalently - the image point is given in particular by the values ​​of two Cartesian coordinates - without loss of generality x and y - assigned to the tip end in the original plane (without loss of generality given by z = 0) of the sensor coordinate system.

[0086] In principle, various algorithms are possible for determining the position of the tip end. In particular, in addition to classic image processing methods, the use of CNNs (convolutional neural networks) or other machine learning-based methods is also possible.

[0087] In one embodiment, a high-contrast image of the pipette tip is used by the optical sensor against a uniform background, particularly against a dark background. Advantageously, the typically transparent pipette tip then stands out brightly and clearly against the background.

[0088] Furthermore, for image evaluation, it is sufficient to use a section of the image or recording - these terms are used synonymously here - of the optical sensor that shows the pipette tip and in particular the tip end.

[0089] In one embodiment, the captured image or image section is converted to a grayscale representation, and Harris Corner detection is performed. This provides a value for each pixel that describes the intensity changes around the pixel. Above a certain threshold of this value, the respective pixel is classified as a Harris Corner pixel. The Harris Corner pixels are then fed to a line detection algorithm that—for example, using the Ransac method (Random Sample Consensus)—filters out lines from the point coordinates of the Harris Corner pixels as the edge lines. Preferably, however, a restriction in the form of an angle specification is also specified, so that, for example, no vertical or horizontal lines in the image plane are recognized as edge lines.In particular, the edge lines are obtained as those straight lines that are closest to the most Harris Corner image points that do not exceed a predetermined maximum distance from the respective edge line. An intersection point of the two detected edge lines is then calculated. Generally, however, this does not yet correspond to the tip end image point, since typical pipette tips do not always converge in two straight lines, but rather, for example, have a different gradient in an attachment area than in the tip end area. Therefore, instead, in a predetermined, close vicinity of the intersection point – which is smaller, in particular much smaller, than the evaluated image section – the Harris Corner image point that exhibits a maximum deflection in the image plane evaluated in the direction of the pipette axis – in particular toward a presumed position of the tip end – is identified as the tip end.This takes advantage of the fact that the image point of the tip end itself is almost always part of the detected Harris Corner image points, at least in the case of a high-contrast background, since the pipette tip has the strongest curvature here.

[0090] Advantageously, the absence of a pipette tip can also be deduced by comparing it with image positions of the tip end determined in the past and / or based on a discontinuity in the detected image position of the tip end.

[0091] According to a further development of the invention, a well of the multiwell plate into which the pipette tip is immersed is detected when the pipette tip crosses a reference plane—in particular, the pipetting surface—in which the upper edges of the wells of the multiwell plate lie. Advantageously, this allows reliable detection of a well into or from which pipetting is currently taking place.

[0092] The well into which the pipette tip is immersed is also called the immersion well.

[0093] In one embodiment, the reference plane is an origin plane of the multi-well plate coordinate system, that is to say, in particular without loss of generality, a plane spanned by the x-coordinate and the y-coordinate in which the following applies: z = 0.

[0094] In one embodiment, the origin of the multiwell plate coordinate system lies at a center point of the pipetting surface of the multiwell plate. Without loss of generality, the x-coordinate and the y-coordinate of the multiwell plate coordinate system span the pipetting surface; the z-coordinate is perpendicular to the pipetting surface, and on the pipetting surface, z = 0. The reference plane is thus the plane in which the pipetting surface lies. Again, without loss of generality, the positive branch of the z-axis extends from the pipetting surface away from the multiwell plate, essentially "upwards" in a direction in which the wells are open, i.e., typically geodesically upwards, while the negative branch of the z-axis extends from the pipetting surface into the multiwell plate, typically geodesically downwards.Depending on the design of the edge of the multiwell plate or the reference frame, the reference markings can be located in a plane that is offset, in particular parallel, to the pipetting surface.

[0095] The fact that the pipette tip crosses the reference plane means, in particular, that the pipette tip crosses the reference plane. In particular, the immersion well is detected when the pipette tip crosses the reference plane from top to bottom, i.e., from positive z-values ​​to negative z-values, which means, in particular, when the z-coordinate of the pipette tip changes sign from positive to negative.

[0096] In particular, the immersion well is detected when the tip end of the pipette tip crosses the reference plane.

[0097] In one embodiment, the position of the pipette tip, in particular the tip end, is continuously evaluated in the multi-well plate coordinate system, with the immersion well being detected at the moment the z-coordinate changes sign. As explained above, it is not absolutely necessary to continuously determine the position of the pipette tip; rather, it can be derived from the preferably continuously determined position of the optical sensor in the multi-well plate coordinate system if the position of the pipette tip relative to the optical sensor in the sensor coordinate system is known with sufficient accuracy. However, it is also possible to continuously evaluate the position of the pipette tip.

[0098] The immersion well is preferably identified by evaluating the x- and y-coordinates of the intersection point of the pipette axis with the reference plane at the moment the z-coordinate changes sign. The well address of the immersion well can then be calculated from the format of the multiwell plate, for example, a 96-, 384-, or 1536-well plate, by dividing the respective coordinate by the (equal) spacing between the wells. This allows the well address of the immersion well currently addressed by the pipette to be recorded. Further information can be determined, including how fast the pipette tip is moved in the immersion well, the immersion depth it reaches, its angular orientation, and other information.

[0099] Advantageously, the determination of the immersion depth can be used in particular to evaluate the quality of the pipetting process with regard to whether the tip end was held exactly in the area of ​​a surface of the pipetted medium in order to minimize, preferably avoid, any influence of the hydrostatic pressure on the pipetting process.

[0100] According to a further development of the invention, a detected immersion well is held until the pipette tip, in particular the tip end, crosses the reference plane again. This advantageously prevents incorrect detection of wells addressed by the pipette; this risk would otherwise exist because the tip end is no longer visible in the immersed state, and extrapolation of the pipette tip in the two-dimensional image of the optical sensor in this situation would typically point to another well adjacent to the immersion well actually to be detected, especially if the pipette tip is slightly bent in the well.

[0101] Holding the detected immersion well means in particular that the coordinates or well address of the immersion well are held – i.e., not changed – until the pipette tip crosses the reference plane again.

[0102] The fact that the pipette tip crosses the reference plane again means, in particular, that the pipette tip now crosses the reference plane in the opposite direction. In particular, the detected immersion well is held until the pipette tip, especially the tip end, crosses the reference plane from bottom to top, i.e., from negative z-values ​​to positive z-values, especially when the z-coordinate of the tip end changes sign from negative to positive. This indicates that the pipette tip has left the immersion well.

[0103] According to a further development of the invention, a pipetting process is recorded. This can advantageously be used for monitoring, documentation, and / or quality control of pipetting processes.

[0104] In one embodiment, the recorded pipetting process is logged. This advantageously provides an automated option for logging a process that is actually manual, which significantly simplifies the corresponding work and allows the operator to concentrate on the more experimentally or laboratory-relevant steps of the pipetting process.

[0105] In one embodiment, when recording the pipetting process, at least one piece of additional information is recorded, which is preferably selected from a group consisting of a pipetted medium, a pipetted volume, an assignment of an actually performed pipetting step to a detected immersion well, a linear acceleration of the pipette, an angular velocity of the pipette, and a combination of at least two of the said pieces of additional information.

[0106] In the context of the present technical teaching, “pipetting” or “pipettes” is understood in particular to mean dispensing or taking up a medium or volume with the pipette.

[0107] It is possible to determine the pipetted volume optically by evaluating the images captured by the optical sensor. However, this is typically only possible with any degree of accuracy if the pipetted medium is colored.

[0108] In one embodiment, alternatively or additionally, at least one of the aforementioned additional information items is captured using an additional sensor, for example, by using a digital pipette connected to the control device via a suitable interface. Alternatively or additionally, an additional sensor can be attached to the pipette, in particular a touch sensor on a pipetting button of the pipette, which can be used, in particular, to determine whether a pipetting step has actually been performed. Optionally, the depth of the pipetting button's depression and thus the pipetted volume can also be captured.

[0109] Recording whether a pipetting step was actually performed is important for creating a correct record of the pipetting process; for example, it could be that the pipette is simply accidentally immersed in a certain well without actually pipetting there.

[0110] The linear acceleration and / or angular velocity of the pipette is preferably detected by means of at least one inertial sensor.

[0111] Finally, alternatively or additionally, the use of additional devices for recording or entering the at least one additional piece of information is also possible, for example by the operator entering or acknowledging corresponding information on a tablet that is operatively connected to the control device, for example via Bluetooth or WiFi.

[0112] In particular, the technical teaching presented here can be used to prevent pipetting errors by ensuring, through warnings or instructions to the operator, that the medium to be pipetted is assigned to the correct wells. Furthermore, the operator can be supported by being guided to adhere to a predetermined pipetting sequence by highlighting target wells. Furthermore, the operator is relieved of potentially overwhelming tasks, such as counting wells and / or, in some cases, remembering the last well used after an interruption. Automatic, detailed recording of the work steps performed leads to improved documentation and traceability of experiments and assays. In particular, data integrity and / or data consistency can be improved by linking digital pipetting data with other instruments or laboratory information systems.Recording the pipette position and orientation, in particular, can provide the operator with feedback on the correct execution of pipetting steps and, for example, indicate excessive acceleration, incorrect immersion depths relative to the media surface, or angles that are too shallow. Digital recording can also serve as a teaching tool to teach operators the correct pipetting technique. By documenting the images from the optical sensor, a dual control principle can be implemented. This enables monitoring, and rules and regulations, particularly with regard to process documentation, for example, in GMP processes, can be met with less effort.

[0113] The invention is explained in more detail below with reference to the drawings, which show: Fig. 1 is a diagram of an embodiment of a pipetting detection device; Fig. 2 a detailed representation of the embodiment of the pipetting detection device according to Fig. 1; Fig. 3 a first detailed illustration of an embodiment of a method for detecting a position of a pipette tip of a pipette relative to a multiwell plate, and Fig. 4 a second detailed representation of the embodiment of the method.

[0114] Fig. 1 shows an illustration of an embodiment of a pipetting detection device 1.

[0115] The pipetting detection device 1 has a plurality of reference markings 5, each of which can be arranged or is arranged in a predetermined, in particular known, marking position relative to a multiwell plate 3. For the sake of clarity, only one reference marking 5 is designated here with the corresponding reference symbol. Each reference marking 5 of the plurality of reference markings 5 ​​has at least one reference point 6, wherein, for example, four corner points of one of the reference markings 5 ​​are designated as reference points 6 with the corresponding reference symbol.In addition, the pipetting detection device 1 comprises an optical sensor 7, which can be arranged or is arranged and configured in a fixed position relative to a pipette 9—in particular in a predetermined, in particular known, sensor position—in such a way that at least a minority of the reference points 6 and preferably simultaneously a pipette tip 11 of the pipette 9 can be optically detected by means of the optical sensor 7. Finally, the pipetting detection device 1 comprises a control device 13, which is operatively connected to the optical sensor 7 and configured to determine a three-dimensional position, preferably a six-dimensional position or orientation of the pipette tip 11 based on the detected reference points 6.

[0116] The multiwell plate 3 has a plurality of wells 12 arranged or formed on or in particular regularly in a rectangular pattern on the multiwell plate 3, of which, for the sake of better clarity, only one is also identified with a corresponding reference symbol.

[0117] In particular, the reference markings 5 ​​can be arranged or are arranged in a fixed position relative to the multiwell plate 3. The optical sensor 7 can be arranged or are arranged in a fixed position relative to the pipette 9.

[0118] The optical sensor 7 is preferably designed as a camera. In one embodiment, the optical sensor 7 is formed in multiple parts with the pipette 9 and can be attached or is attached to the pipette 9. In the exemplary embodiment shown here, however, the optical sensor 7 is integrated into the pipette 9 or arranged as a single piece thereon. Thus, the pipette 9 is also part of the pipetting detection device 1.

[0119] In particular, the optical sensor 7 is arranged on a pipette base body 15 of the pipette 9, preferably formed in one piece with the pipette base body 15 or integrated into the pipette base body 15.

[0120] The pipette 9 is preferably designed as a hand-held pipette 9.

[0121] The control device 13 is preferably integrated into the pipette 9 or arranged on the pipette 9. Alternatively, however, it is also possible—as shown here, however, essentially for the sake of simplicity—for the control device 13 to be formed externally to the pipette 9. The control device 13 is operatively connected, in particular, to the optical sensor 7—and optionally additionally to other sensors of the pipette 9—via a cable connection or wirelessly.

[0122] The pipette base body 15 preferably has a pipette tip receptacle 17, on which the pipette tip 11 can be exchangeably arranged. The pipette 9 is designed in particular as a graduated micropipette, for example, as an Eppendorf pipette.

[0123] The pipetting detection device 1 preferably has a display device 19, which is also shown only schematically here for the sake of simplicity and is external to the pipette 9. The display device 19 is operatively connected to the control device 13 and configured to display at least one piece of information relevant to a pipetting process to be performed. The display device 19 can be configured to output the relevant information visually, acoustically, haptically, and / or in the form of a vibration. In particular, the display device 19 can—additionally or alternatively—also be configured to cause the pipette 9 to vibrate when the pipette tip 11 is arranged over a well of the wells 12 to be addressed.In particular, the display device 19—in combination with the control device 13—is configured to display the information relevant to the pipetting process to be performed, depending on the determined position of the pipette tip 11, and / or to display or output coordinates of a well 12 to be addressed next. The display device 19 can be designed as a screen separate from the pipette 9 and the multiwell plate 3, or as a headset or glasses, in particular as a head-up display worn on the body. Furthermore, the display device 19 can be designed as a display device attached to the pipette 9, formed integrally with the pipette 9, or integrated into the pipette 9, in particular as a monitor or as a head-up display carried along with the pipette 9.In yet another embodiment, the display device 19 can have a plurality of separate light sources, preferably LEDs, arranged, for example, in the form of a coordinate system or in the form of a two-dimensional matrix, by means of which information can be displayed, in particular individual wells 12 of the multi-well plate 3 can be highlighted or illuminated. In particular, the display device 19 configured in this way can be arranged to be arranged below the then preferably transparent or translucent multi-well plate 3, so that the light sources are visible through the multi-well plate 3 and in particular through the individual wells 12, so that individual wells 12 can be highlighted by controlling the correspondingly - preferably uniquely - assigned light sources.In yet another embodiment, the display device 19 may comprise a projection device or be designed as a projection device, wherein the projection device is configured to project information, in particular an image or character, onto the multi-well plate 3 and thus in particular to mark or highlight individual wells 12.

[0124] The pipetting detection device 1 preferably has at least one inertial sensor 21, in particular an inertial sensor, which can be arranged or is arranged on the pipette 9, in particular integrated into the pipette 9, and which is operatively connected to the control device 13. With the aid of the at least one inertial sensor 21, the movement of the pipette 9 can advantageously be detected even more precisely, wherein, in particular, a displacement of the pipette 9 can even be further tracked and / or extrapolated into a situation in which fewer than three reference points 6 or no reference point 6 can be detected.

[0125] The control device 13 is particularly configured to carry out a method described below.

[0126] Fig. 2 shows a detailed representation of the embodiment of the pipetting detection device 1 according to Fig. 1.

[0127] Identical and functionally identical elements are provided with the same reference symbols in all figures, so that reference is made to the preceding description in each case.

[0128] In the exemplary embodiment shown here, the reference markings 5 ​​are not arranged directly on the multiwell plate 3, but on a reference frame 23 as an additional component, wherein the reference frame 23 is in turn arranged in a stationary manner, in particular in a fixed position, relative to the multiwell plate 3. In another exemplary embodiment not shown, the reference markings 5 ​​can also be arranged directly on the multiwell plate 3, for example, printed, embossed into the multiwell plate 3, formed in relief on the multiwell plate 3, formed as a surface structure of the multiwell plate 3, glued on, or attached in some other way.

[0129] In the embodiment shown here, the reference markings 5 ​​are arranged completely along a closed circumferential line of the multiwell plate 3, as well as in the area of ​​a pipetting surface 25 of the multiwell plate 3, towards which the wells 12 are open. However, the reference markings 5 ​​are located here (in Fig. 1) in a plane offset parallel to the pipetting surface 25 - towards positive z-values. In the exemplary embodiment shown here, a width of the reference frame 23 is locally adapted to a width of an edge of the multiwell plate 3 that is hidden from the observer, in particular in order to influence the dimensions of the multiwell plate 3 as little as possible, preferably minimally, by attaching the reference frame 23 - particularly with regard to upstream or downstream processing steps. The size of the individual reference markings 5 ​​is selected to be maximum with regard to the width of the reference frame 23 and the preferably square basic shape of the reference markings 5; in general, the dimensions of the reference markings 5 ​​are preferably selected such that, on the one hand, the best possible optical recognizability is ensured and, on the other hand, as many reference markings 5 ​​as possible can be arranged along the closed circumferential line.In particular, the sizes of the reference markings 5 ​​are known.

[0130] The reference frame 23 is designed to be arranged in a fixed position, in particular in a fixed position, on the multiwell plate 3. Preferably, the reference frame 23 can be placed or slipped over the multiwell plate 3 in such a way that it is connected to it in a fixed position, in particular in a fixed position, in particular in a form-fitting manner. Alternatively, the multiwell plate 3 can be inserted into the reference frame 23, in particular from above.

[0131] The reference markings 5 ​​can be printed on the reference frame 23, embossed into the reference frame 23, formed raised on the reference frame 23, formed as a surface structure of the reference frame 23, or otherwise stably connected thereto, in particular fastened thereto, for example glued onto the reference frame 23.

[0132] In the exemplary embodiment illustrated here, the reference markings 5 ​​are each embodied as a two-dimensional code, with each reference marking being uniquely, in particular uniquely, assigned such a code. The two-dimensional code is preferably selected from a group consisting of AprilTag, ArUco-Tag, ARTag, ARToolKit, Intersense, RUNE-43, RUNE-129, reacTIVIsion, Framedge, and QR code. In the specific exemplary embodiment illustrated here, the two-dimensional code is AprilTag.

[0133] Fig. 3 shows a first detailed illustration of an embodiment of a method for detecting a position of the pipette tip 11 relative to the multiwell plate 3.

[0134] Within the scope of the method, a plurality of the reference points 6 arranged in the respective predetermined positions relative to the multi-well plate 3 and preferably at the same time the pipette tip 11 - in particular an outermost, distal tip end 27 of the pipette tip 11 - are detected by the optical sensor 7, and the position of the pipette tip 11 is determined based on the detected reference points 6.

[0135] In particular, a perspective projection, in particular based on the model of a pinhole camera, is used to determine the position of the pipette tip 11. The position of the pipette tip 11 is preferably determined using homogeneous coordinates, in particular augmented by a scaling coordinate. In particular, points in three-dimensional space are projected onto a two-dimensional plane, in particular an image plane of the optical sensor 7, using a projection matrix that describes the internal and external orientation of the optical sensor 7.Conversely, the position and orientation of the recording optical sensor 7 can be determined from known points in three-dimensional space—specifically, the reference points 6 at the marking positions—and their corresponding coordinates in the two-dimensional image, provided its calibration matrix is ​​known and a minimum number of known image points, in particular three image points, are present. Preferably, a perspective-n-point problem (PnP problem) is solved based on the acquired reference points 6. This solution then provides, in particular, the position and pose of the optical sensor 7 relative to the multi-well plate coordinate system.

[0136] In Fig. 3 specifically shows that a distance between the pipette tip 11 - in particular the tip end 27 - and the optical sensor 7 is calibrated by placing the tip end 27 on the reference frame 23 with slight force in a position determined by at least one reference marking 5 of the plurality of reference markings 5, wherein the distance is determined based on the at least one reference marking 5.

[0137] Preferably, the distance is determined based on the at least one reference marking 5 and an image position of the tip end 27 in an image captured by the optical sensor 7. The image is captured when the tip end 27 is placed in the position determined by the at least one reference marking 5.

[0138] A pipette axis 29, which runs centrally through the pipette base body 15 to the tip end 27, is described in this approach by means of two points p1, p2. A first point p1 lies in an image plane of the optical sensor 7, which is used as the origin plane, in particular the x,y plane at z = 0, of a sensor coordinate system. The xy coordinate of the first point p1 is known from the predetermined, known sensor position, i.e. in particular the known dimensions of a sensor adapter or the known placement of the integrated optical sensor 7 on the pipette base body 15. The distance of the optical sensor 7 from the pipette axis 29 is rigid and does not normally change; therefore, the first point p1 for the pipette axis 29 can be defined in the sensor coordinate system. The position of a second point p2 of the pipette axis 29 is determined by the image point, i.e. an image position, of the pipette tip 27 in the image of the optical sensor 7 - iein the xy plane at z = 0 of the sensor coordinate system - and the depth information determined from the at least one reference marking 5. Starting from the image point alone, the pipette tip 27 can theoretically lie on an infinitely long line at a specific angle to the optical sensor 7. However, since the distance from the pipette tip 27 to the optical sensor 7 is precisely known from the depth information, the position of the pipette tip 27 can be calculated using the known length and two known angles in the vertical and horizontal image planes. Thus, the second point p2 of the pipette axis 29 is also known, and the complete position of the pipette tip 27 in space can be determined with 6 degrees of freedom.

[0139] Based on this calibration, the current position of the pipette axis 29 in space can now be determined during a pipetting process. This allows for the detection and recording of impermissible angular alignments during pipetting, such as an excessively shallow angle. Alternatively or additionally, a warning can be issued if the pipette axis 29 is held at an impermissible angle, for example, visually, acoustically, haptically, or by vibration of the pipette 9.

[0140] Fig. 4 shows a second detailed representation of the embodiment of the method.

[0141] Based on Fig. Figure 4 explains in particular how the image position of the outermost tip end 27 in a recording of the optical sensor 7, i.e., the image point, is determined. For this purpose, edge lines 31 of the pipette tip 11 are detected in the recording, and the image position of the tip end 27 is determined based on the detected edge lines 31.

[0142] For image evaluation, an image section of the recording is preferably used, wherein the image section shows the pipette tip 11 and in particular the tip end 27.

[0143] Preferably, the image section is converted to a grayscale representation, and Harris Corner detection is performed. This provides a value for each pixel in the image section that describes the intensity changes around the respective pixel. Above a certain threshold value of this value, the pixel is classified as a Harris Corner pixel. The Harris Corner pixels are then fed to a line detection algorithm, which—for example, using the Ransac method (Random Sample Consensus)—filters lines from the point coordinates of the Harris Corner pixels as the edge lines 31.

[0144] Preferably, a restriction in the form of an angle specification is also specified, so that, for example, no vertical or horizontal lines in the image plane are recognized as the edge lines 31. In particular, the edge lines 31 are obtained as those straight lines that have the shortest distance to the most Harris Corner image points that do not exceed a predetermined maximum distance from the respective edge line 31. An intersection point S of the recognized edge lines 31 is then calculated. However, this intersection point does not usually correspond to the image point of the tip end 27, since typical pipette tips 11 do not always run in two straight lines, but, for example, have a different gradient in an attachment area than in the area of ​​the tip end 27.Therefore, instead, in a predetermined, close vicinity U of the intersection point S – which is smaller, in particular much smaller, than the evaluated image section – the Harris Corner image point is identified as the image point of the tip end 27 that exhibits a maximum deflection in the (distal) direction of the pipette axis 29 in the image plane – in particular toward the presumed position of the tip end 27. This takes advantage of the fact that the image point of the tip end 27 itself is almost always part of the detected Harris Corner image points, at least in the case of a high-contrast background, since the pipette tip 27 exhibits the greatest curvature at this point.

[0145] Advantageously, an absence of a pipette tip 11 can also be deduced by comparing it with image positions of the tip end 27 determined in the past and / or based on a discontinuity of the detected image position of the tip end 27.

[0146] Within the scope of the method, it is further preferably provided that an immersion well of the wells 12 of the multiwell plate 3, into which the pipette tip 11 is immersed, is detected when the pipette tip 11 crosses a reference plane—in particular defined by the pipetting surface 25—in which the upper edges of the wells 12 of the multiwell plate 3 lie. Preferably, the reference plane is an origin plane of the multiwell plate coordinate system, i.e., in particular, a plane spanned by the x-coordinate and the y-coordinate, in which z = 0 applies.

[0147] In one embodiment, the origin of the multiwell plate coordinate system lies at a center point of the pipetting surface 25 of the multiwell plate 3. Without loss of generality, the x-coordinate and the y-coordinate of the multiwell plate coordinate system span the pipetting surface 25; the z-coordinate is perpendicular to the pipetting surface 25 and on the pipetting surface 25 the following applies: z = 0. Again without loss of generality, the positive branch of the z-axis extends from the pipetting surface 25 away from the multiwell plate 3, so to speak "upwards" in a direction in which the wells 12 are open, i.e. typically geodesically upwards, while the negative branch of the z-axis extends from the pipetting surface 25 into the multiwell plate 3, typically geodesically downwards.

[0148] In particular, the immersion well is detected when the tip end 27 crosses the reference plane from top to bottom, i.e., from positive z-values ​​to negative z-values, i.e., particularly when the z-coordinate of the tip end 27 changes sign from positive to negative. Preferably, the position of the tip end 27 in the multiwell plate coordinate system is continuously evaluated, with the immersion well being detected at the moment of the change in sign of its z-coordinate.

[0149] The immersion well is preferably detected by evaluating the x- and y-coordinates of the intersection point of the pipette axis 29 with the reference plane at the moment the z-coordinate of the tip end 27 changes sign. The well address of the immersion well can then be calculated from the format of the multiwell plate 3, for example, a 96-well, 384-well, or 1536-well plate, by dividing the respective coordinate by the (uniform) spacing of the wells 12. Thus, the well address of the immersion well currently addressed by the pipette 9 can be recorded. Further information that can be determined includes how fast the pipette tip 11 is moved in the immersion well, the immersion depth it reaches, its angular orientation, and other information.

[0150] Preferably, a detected immersion well is held until the tip end 27 crosses the reference plane again. In particular, the detected immersion well is held until the tip end 27 crosses the reference plane from bottom to top, i.e., from negative z-values ​​to positive z-values, i.e., in particular, when the z-coordinate of the tip end 27 changes sign from negative to positive. This indicates that the tip end 27 has left the immersion well.

[0151] Preferably, within the scope of the method, a pipetting process is recorded, and the associated pipetting procedure is monitored, documented, and / or subjected to quality control. Preferably, the recorded pipetting process is logged. Preferably, when recording the pipetting process, at least one piece of additional information is also recorded, which is in particular selected from a group consisting of a pipetted medium, a pipetted volume, an assignment of an actually performed pipetting step to a detected immersion well, a linear acceleration of the pipette 9, an angular velocity of the pipette 9, and a combination of at least two of the aforementioned pieces of additional information.

[0152] In this case, a digital pipette 9 can be used, which is connected to the control device 13 via a suitable interface, or an additional sensor can be attached to the pipette 9, in particular a touch sensor on a pipetting button 33 (see Fig. 1) of the pipette 9, which can be used, in particular, to determine whether a pipetting step has actually been performed and, if necessary, also to detect the insertion depth of the pipette button 33 and thus the pipetted volume. The linear acceleration and / or angular velocity of the pipette is preferably detected by the inertial sensor 21.

[0153] It is also possible to use additional devices for recording or entering the at least one additional piece of information, for example by the operator entering or acknowledging corresponding information on a tablet that is operatively connected to the control device 13, for example via Bluetooth or WiFi. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] DE 10 2020 208 740 A1

[0002]

Claims

[1] Pipetting detection device (1) with a plurality of reference markings (5) which can be arranged or are arranged in a fixed position relative to a multi-well plate (3) in each case in a predetermined marking position, wherein each reference marking (5) of the plurality of reference markings (5) has at least one reference point (6), an optical sensor (7) which can be arranged or is arranged and configured in a fixed position relative to a pipette (9) in such a way that at least a minority of the reference points (6) can be optically detected by means of the optical sensor (7), and with a control device (13) which is operatively connected to the optical sensor (7) and is configured to determine a three-dimensional position of the pipette tip (11) on the basis of the detected reference points (6). [2] Pipetting detection device (1) according to claim 1, wherein the reference markings (5) are arranged on a reference frame (23) which is optionally designed to be arranged in a fixed position, in particular in a fixed position, on a multi-well plate (3). [3] Pipetting detection device (1) according to one of the preceding claims, wherein the pipetting detection device (1) further comprises the pipette (9), wherein the optical sensor (7) is arranged on a pipette base body (15) of the pipette (9), in particular fastened to the pipette base body (15), integrated into the pipette base body (15) or formed in one piece with the pipette base body (15). [4] Pipetting detection device (1) according to claim 3, wherein the pipette base body (15) has a pipette tip receptacle (17) on which the pipette tip (11) can be arranged in an exchangeable manner. [5] Pipetting detection device (1) according to one of the preceding claims, with a display device (19) which is operatively connected to the control device (13) and is designed to display at least one item of information relevant to a pipetting process to be carried out, in particular depending on the determined position of the pipette tip (11). [6] Pipetting detection device (1) according to one of the preceding claims, wherein the reference markings (5) are each formed as a two-dimensional code, wherein the two-dimensional code is in particular selected from a group consisting of AprilTag, ArUco-Tag, ARTag, ARToolKit, Intersense, RUNE-43, RUNE-129, reacTIVIsion, Framedge, and QR code. [7] Pipetting detection device (1) according to one of the preceding claims, wherein the control device (13) is arranged to carry out a method according to one of claims 8 to 16. [8] Method for detecting a three-dimensional position of a pipette tip (11) of a pipette (9) relative to a multiwell plate (3), comprising the following steps: - optically detecting a plurality of reference points (6) arranged in predetermined positions relative to the multi-well plate (3) by an optical sensor (7) arranged in a fixed position on the pipette (9), wherein the reference points (6) are assigned to a plurality of reference markings (5), wherein each reference marking (5) of the plurality of reference markings (5) has at least one reference point (6) of the plurality of reference points (6), and - Determining the three-dimensional position of the pipette tip (11) based on the recorded reference points (6). [9] Method according to claim 8, wherein additionally the pipette tip (11) is detected by the optical sensor (7). [10] Method according to one of claims 8 or 9, wherein a two-dimensional code is used as the reference markings (5), wherein the two-dimensional code is in particular selected from a group consisting of AprilTag, ArUco-Tag, ARTag, ARToolKit, Intersense, RUNE-43, RUNE-129, reacTIVIsion, Framedge, and QR code. [11] Method according to one of claims 8 to 10, wherein a perspective projection is used to determine the position of the pipette tip (11), preferably a perspective-n-point problem being solved on the basis of the detected reference points (6). [12] Method according to one of claims 8 to 11, wherein a distance between the pipette tip (11) and the optical sensor (7) is determined by placing the pipette tip (11) in a position determined by at least one reference marking (5) of the plurality of reference markings (5), the distance being determined based on the at least one reference marking (5). [13] Method according to one of claims 9 to 12, wherein an image position of an outermost tip end (27) of the pipette tip (11) is determined in a recording of the optical sensor (7) by detecting edge lines (31) of the pipette tip (11) in the recording and determining the image position of the tip end (27) based on the detected edge lines (31). [14] Method according to one of claims 8 to 13, wherein a well (12) of the multi-well plate (3) into which the pipette tip (11) is immersed is detected when the pipette tip (11) - in particular the outermost tip end (27) of the pipette tip (11) - crosses a reference plane in which upper edges of the wells (12) of the multi-well plate (3) lie. [15] The method of claim 14, wherein a detected well (12) into which the pipette tip (11) dips is held until the pipette tip (11) crosses the reference plane again. [16] Method according to one of claims 8 to 15, wherein a pipetting process is recorded and preferably logged.

Images