Devices, systems, and methods for manipulating sample containers

Abstract

According to the invention, a robotic handling device for handling sample containers is provided in which the sample containers are arranged or to be arranged within a sample container holder. The robotic handling device includes: a gripping element for gripping and releasing a sample container; a driving mechanism configured to move the gripping element; and at least one sensing device connected to the gripping element wherein the at least one sensing device is configured to collect sensing data relating to the sample container holder, the sensing data comprising optical imaging data and ranging data; wherein the drive mechanism is configured to move the gripping element based on identification data relating to the sample container holder, the identification data being obtained from the sensing data.

Description

Technical Field

[0001] The present invention relates to an apparatus, a system and a method for detecting a sample container arranged in a sample container holder and for gripping a sample container from a sample container holder or releasing a sample container into a sample container holder. Background Art

[0002] Sample containers are used to hold samples to be tested, such as biological samples, such as blood samples, extracted from human or animal patients. Typically, biological samples are collected at a first location, such as by a phlebotomist, and then transported to a second location, such as a laboratory, for testing / analysis. Sample container holders (or "holders") are used to hold one or more sample containers, such as tubes, particularly to ensure that the sample tubes remain in a generally upright orientation, such as during transport.

[0003] Typically, the sample container rack is automatically manipulated at the second position. For example, when the sample container rack is located at, for example, an input station of an automated laboratory system, the robot can pick up the sample container from the rack according to a predetermined regular order, for example, starting from the upper left corner of the rack. When the sample container rack is located at, for example, an output station of an automated laboratory system, the robot can simply assume that the rack is empty and fill the rack according to a predetermined regular order. In other words, the robot can manipulate each rack according to the same fixed rules that dictate the robot's movements. Summary of the invention

[0004] It is an object of the present invention to increase throughput and minimize errors when handling sample containers containing samples to be analyzed.

[0005] The achievement of this object according to the invention is set forth in the independent claim. Further developments of the invention are the subject matter of the dependent claims.

[0006] According to a first aspect of the present invention, there is provided a robot manipulator for manipulating a sample container, wherein the sample container is arranged or to be arranged in a sample container holder. The robot manipulator comprises:

[0007] - a gripping element for gripping and releasing the sample container;

[0008] - a drive mechanism configured to move the clamping element; and

[0009] - at least one sensing device connected to the clamping element, wherein the at least one sensing device is configured to collect sensing data related to the sample container holder, the sensing data comprising optical imaging data and ranging data;

[0010] Therein, the drive mechanism is configured to move the gripping element based on identification data associated with the sample container holder, the identification data being obtained from the sensing data.

[0011] Typically, a sample container is configured to receive and hold a sample, such as a biological sample. Thus, a sample container may include a vessel comprising an opening for receiving the sample, wherein the opening may be configured to be releasably closed by a closure element, such as a cover. Thus, a sample container may include a vessel and a cover that optionally releasably seals the opening of the vessel.

[0012] In particular, the vessel can be a test tube with a generally cylindrical shape, the test tube having an opening at one base and a closed end at an opposite base, the test tube having a given length given by the height of the cylindrical member. Exemplarily, the longitudinal cross-section of the closed end can be U-shaped. The cover can include an engaging portion configured to engage with the vessel (e.g., configured to be pushed into the vessel to form a shape-fitting connection with at least a portion of the vessel) and a protruding portion configured to protrude above the vessel. Exemplarily, the protruding portion can be a hollow cylindrical portion having an annular cross-section (also referred to as a "top ring"). Alternatively, the cover can include a recess for receiving a portion of the vessel and forming a shape-fitting connection with a portion of the vessel.

[0013] A biological sample can be a sample of a body fluid of a human or animal subject. For example, a body fluid can be a physiological fluid, such as blood, saliva, urine, sweat, amniotic fluid, cerebrospinal fluid, ascites, etc. The biological sample can be placed in a sample container after collection, and the biological sample can be (at least partially) maintained in the container during processing. In addition to the biological sample, the sample container can contain one or more substances, such as reagents.

[0014] The sample container holder is configured to receive and hold at least one sample container, in particular to receive and hold a plurality of sample containers. Thus, the holder may comprise a plurality of receiving portions, wherein each receiving portion is configured to hold a corresponding sample container, in particular to hold it in a substantially upright position. The receiving portion may be a recess or an opening.

[0015] The support may include a plate, the plate including a plurality of openings, wherein each opening is configured to receive a corresponding sample container. Thus, the receiving portion may include or consist of an opening. Exemplarily, the opening may be circular. A plurality of receiving portions may be arranged in a grid formed by rows and columns. For example, the support may have 6×6 receiving portions or 6×2 receiving portions. In particular, the plate may be referred to as an “upper plate” and the opening may be referred to as an “upper opening”, because when the support is used in conjunction with a robotic manipulator, for example when the support is placed on a surface, the plate and its opening constitute an upper portion of the support, for example the plate is positioned above the rest of the support.

[0016] The support can also include a supporting element, which is configured to support the upper plate above the surface on which the support is placed, particularly so that the upper plate is located at a given height from the surface. This height can make the upper part of the sample container be arranged and maintained above the support (i.e., outside the support). In other words, when being maintained in the support, the upper part of the sample container protrudes beyond the support and can be gripped by the clamping element. The upper part of the sample container includes an opening, which is optionally sealed by a lid. In the example of a test tube, the upper part can include 20% of the tube length or 10% of the tube length or 5% of the tube length. Also as explained above, the "upper part" of the sample container refers to the part of the sample container located above the rest of the sample container when the sample container is in use, such as operated by a robot manipulator or maintained in the support.

[0017] For example, the support element can be two lateral plates, for example perpendicular to the upper plate and located on two opposite sides of the upper plate. Thus, in one example, the bracket can have the shape of a c-shaped bracket. In another example, the bracket can be substantially shaped like a cuboid and / or a box, and also includes a support plate parallel to the upper plate and connected to the lateral plates. Optionally, the bracket can also include a plurality of plates intersecting to form a prismatic unit, each unit corresponding to an orifice. In this case, the receiving portion can also include a lateral wall given by the unit.

[0018] The robot manipulator is configured to manipulate sample containers, in particular sample containers that have been arranged in the support (i.e., held in the receiving portion of the support) and sample containers to be arranged in the support, i.e., to be received by the support, for example, to be inserted in the corresponding receiving portion of the support. The robot manipulator is configured to move the sample container in three-dimensional space, i.e., change the position of the sample container and / or obtain information about the sample container. In particular, the three-dimensional space is a physical environment in which the robot manipulator, the support and the sample container exist or contain. Therefore, if the position of the sample container changes and / or information about the sample container is obtained by the robot manipulator, the sample container is manipulated by the robot manipulator.

[0019] Specifically, the robot manipulator includes a gripping element configured to grip the sample container, i.e., to grip the sample container so that the sample container is securely held and can move with the gripping element, and to release the sample container, i.e., to release the sample container from its grip and allow physical separation between the gripping element and the sample container. For example, the gripping element can be configured to hold the upper portion of the sample container.

[0020] In particular, the gripping element can be configured to load and unload the sample container rack, i.e. to place the sample container into the rack, e.g. into a corresponding receiving portion of the rack, and to remove the sample container from the rack, e.g. from a corresponding receiving portion of the rack, respectively. When loading, the gripping element can grip the sample container from a loading position, e.g. from a carrier, and release the sample container into the rack, while when unloading, the gripping element can grip the sample container from the rack and release the sample container at an unloading position, e.g. into the carrier. The loading / unloading position can also be, e.g., a receiving portion in the same rack or another rack.

[0021] The gripping element may be configured to grip / release one sample container at a time. In other words, the robot manipulator may manipulate a plurality of different sample containers one at a time, for example in a time sequence. As mentioned, the sample container may be arranged in a support or may have to be arranged in a support.

[0022] Exemplarily, the clamping element may comprise a body and a plurality of clamping fingers, i.e. at least two clamping fingers. The clamping fingers may be identical to one another. The clamping fingers may be elongated portions protruding from the body, for example in the shape of a round bracket or an angle bracket. Thus, the clamping fingers may have one end connected to the body (e.g., to the underside of the body) and another end in free contact with the sample container.

[0023] The gripping fingers may be configured to move relative to each other. To grip the sample container, the gripping fingers may be configured to switch together from a first position or open position to a second position or closed position in which the linear spacing between the free ends (particularly between each pair of free ends) is at its maximum, in which the gripping fingers engage the sample container. To release the sample container, the gripping fingers may be configured to switch together from the second position to the first position. The second position may vary depending on the sample container. In the first position and the second position, the free ends of the gripping fingers may be positioned along a first circumference and a second circumference, respectively, and may be angularly equidistant from each other, i.e. if there are n gripping fingers, the angular distance may be 360° / n. The diameter of the first circumference is greater than the diameter of the sample container (e.g., a tube), and the diameter of the second circumference at the second position may coincide with the diameter of the sample container (e.g., a tube).

[0024] The robotic manipulator also includes a drive mechanism configured to move the gripping element. In particular, the drive mechanism can be configured to move the gripping element as a whole (body and finger-like portion together) in three-dimensional space. For example, the drive mechanism can move the gripping element to a location where a sample container is located (e.g., a receiving portion in a tube holder) to grip the sample container, and then move the gripping element to an unloading location for the sample container (e.g., a carrier) to release the sample container.

[0025] Exemplarily, if a Cartesian coordinate system having an x-axis, a y-axis, and a z-axis is assigned to the three-dimensional space, the drive mechanism can be configured to translate the gripping element parallel to any of the axes, so that the movement of the gripping element can be a composite of translations parallel to the three axes. For example, the drive mechanism can include a robotic arm having at least two degrees of freedom.

[0026] In particular, in operation, the z-axis can be a vertical axis, i.e., parallel to the direction of the earth's gravity, and the x-axis and y-axis can define a horizontal plane. Exemplarily, the relative positions of the robot manipulator and the sample container holder during operation can be such that the upper surface of the holder is parallel to the horizontal plane. For example, the holder can be positioned above a holding element, such as a drawer, and the surface of the holding element can be parallel to the horizontal plane.

[0027] In both the first position and the second position, the clamping finger can extend substantially along the z-axis, i.e. the end connected to the body can be located above the free end. In particular, in operation, the aforementioned first circumference and the second circumference are located in a plane substantially orthogonal to the z-axis and substantially parallel to the horizontal plane.

[0028] The drive mechanism and the clamping fingers may be controlled by a computing device, such as the first computing device or the second computing device discussed below.

[0029] The robot manipulator also includes at least one sensing device connected to the clamping element. In particular, the at least one sensing device can be integrated with the clamping element or can be attached to the clamping element. At least one sensing device and the clamping element can be connected to form a substantially rigid body so that the movement of the clamping element causes the same movement of the at least one sensing device. In particular, the at least one sensing device moves together with the clamping element (in particular with the body of the clamping element), which means that when the drive mechanism moves the clamping element, the drive mechanism moves the at least one sensing device together with the clamping element. In some examples, the at least one sensing device moves together with the clamping element in the xy plane, and the at least one sensing device can be fixed along the z-axis.

[0030] In particular, at least one sensing device may be positioned close to the body of the clamping element. As used herein, the term close may refer to a physical relationship including at, on and near. For example, at least one sensing device may be located at the clamping element, specifically on the body, in particular between a plurality of clamping fingers (the ends of the connection of a plurality of clamping fingers), such as at the lower side of the body. In this case, the distance between the at least one sensing device and the clamping fingers, in particular the free ends of the clamping fingers, may be substantially only along the z-axis. In this case, the spatial offset between the at least one sensing device and the clamping element in the xy plane may be of the order of millimeters.

[0031] In another case, the at least one sensing device can be located near the clamping element, in particular near the body, in particular fixedly connected to the body, for example fixedly connected to the body by a rod. In this case, the distance between the at least one sensing device and the clamping finger, in particular the free end of the clamping finger, can be along at least one of the x-axis and the y-axis and along the z-axis. For example, the spatial offset between the at least one sensing device and the clamping element in the xy plane can be between about 6 cm and about 12 cm, in particular between about 8 cm and about 10 cm. In some examples, the sensing device and the clamping element can be in contact with each other. In particular, the at least one sensing device can be calibrated, for example, according to the calibration procedure described below.

[0032] At least one sensing device is configured to collect sensing data associated with the support. The sensing data associated with the support can be associated with one or more receiving portions of the support, wherein each receiving portion can be empty or occupied by a sample container. If the support accommodates one or more sample containers, the sensing data associated with the support can also be associated with one or more sample containers. In other words, the sensing data associated with the support can be associated with one or more empty receiving portions of the support and / or with one or more sample containers arranged in the support (occupied receiving portions of the support). In particular, the sensing data associated with the support can be associated with a part of the support, such as a part of the upper plate of the support.

[0033] For example, if the rack is empty, the sensed data may be associated only with one or more of the empty receptacles of the rack. If the rack is partially occupied by a sample container, the sensed data may be associated with one or more of the empty receptacles of the rack and / or with one or more sample containers held by the rack. If the rack is full, the sensed data may be associated only with one or more sample containers held by the rack.

[0034] Exemplarily, at least one sensing device may be associated with a sensing area, wherein the sensing area is a three-dimensional volume. In particular, the at least one sensing device may be configured to collect sensing data related to the stent when at least a portion of the stent is positioned within the sensing area. The shape and size of the sensing area and the position and orientation of the sensing area relative to the at least one sensing device may depend on the technical specifications and / or sensing characteristics of the at least one sensing device. The at least one sensing device may face the upper plate of the stent from above. In particular, the sensing area moves together with the at least one sensing device and therefore with the clamping element. For example, the sensing area may be a limited portion of a solid angle, such as a spherical sector, the vertex of which is located at a point of the sensing device.

[0035] The sensing characteristics of the sensing device may include one or more of the following: an angular sensing range, a linear sensing range, an orientation sensing range, and a blind spot. The angular sensing range may refer to an angle describing a portion of a solid angle within which one or more sensing devices are configured to collect sensing data. The linear sensing range refers to a distance from a sensing device, wherein the sensing device can only collect sensing data for objects within the distance. A blind spot may be a volume where the sensing device cannot collect sensing data even if it is within the three sensing ranges. It should be noted that the sensing characteristics of the sensing device may vary, for example, depending on environmental conditions such as lighting.

[0036] In addition, at least one sensing device can be configured to collect sensing data related to the support when at least a portion of the support is within the orientation sensing range, i.e., oriented in one or more sensing orientations. For example, one or more sensing orientations can be clearly determined by three Euler angles. Exemplarily, in use, the sensing device can be configured to collect sensing data when the support is oriented so that an ideal plane containing the upper surface of the support is substantially perpendicular to the vertical axis.

[0037] In the case of multiple sensing devices, each sensing device may have its own sensing area, and the intersection of these individual sensing areas may provide an overall sensing area for the multiple sensing devices.

[0038] In one example, at least one sensing device may be configured to collect sensing data associated with the entire support at one time. In another example, at least one sensing device may be configured to collect sensing data associated with only one section of the support at the same time, wherein the section of the support includes a suitable subset of multiple receiving portions of the support. For example, the section of the support may be a portion of the support that is located within the sensing area. A suitable subset may include one or more receiving portions, which may or may not have a sample container arranged therein. For example, a suitable subset may include a receiving portion and its four first neighbors, i.e., two adjacent neighbors in the same row and two adjacent neighbors in the same column. For a receiving portion at an edge, the first neighbors will be three, and for a receiving portion at a corner, the first neighbors will be two.

[0039] For example, at least one sensing device can be configured to collect sensing data by scanning the support, for example, by changing the position of at least one sensing device above the support so that the sensing area includes different sections of the support at each position to collect sensing data. In particular, the drive mechanism can be configured to move a clamping element (to which at least one sensing device is rigidly connected) so that at least one sensing device can collect sensing data related to multiple sections of the support. The multiple sections are different from each other, but can overlap each other, that is, can have one or more receiving portions in common. The sensing data accumulated over time by the at least one sensing device may or may not cover the entire support.

[0040] In summary, at least one sensing device may collect sensing data related to the entire stent at once or stepwise, for example segment by segment. Alternatively, at least one sensing device may collect sensing data related to one or more segments of the stent (but not the entire stent).

[0041] The sensing data collected by at least one sensing device includes optical imaging data and ranging data. In particular, the optical imaging data and the ranging data are related to the same object. In other words, for a given object, both the optical imaging data and the ranging data are collected by at least one sensing device.

[0042] The optical imaging data may include one or more digital images (e.g., color images, grayscale images, black and white images) of a support including a potential sample container. For example, the optical imaging data may include one or more raster images. A raster image includes a set of pixels, such as a two-dimensional array of pixels. Each pixel may be associated with a value representing the gray level of the pixel. Alternatively, each pixel of the image may be associated with multiple values, such as three or four values, representing a color in a color model, such as an RGB color model, a RYB color model, a CMY color model, a CMYK color model, etc. In particular, each of the one or more digital images may be a corresponding frame of a video. If this is the case, the optical imaging data may include a portion of a video.

[0043] The ranging data may include one or more distance measurements of the distance between at least one sensing device and any point of the support (including potential sample containers). For example, the distance measurements may be stored in the form of one or more distance heat maps. For example, the ranging data may include one or more groups of pixels, such as each group of pixels corresponding to a distance heat map, wherein each pixel may be associated with a value of the distance measurement.

[0044] Exemplarily, at least one sensing device may include an optical camera device (i.e., a camera device that uses visible light or infrared light to create images and / or videos) and a rangefinder device, such as a LIDAR device. Exemplarily, the optical camera device is a camera and / or a video camera. The optical camera device may be configured to collect optical imaging data, and the rangefinder device may be configured to collect ranging data. In particular, the optical camera device may be associated with a first field of view, and the rangefinder may be associated with a second field of view. In this case, the sensing area may be a volume derived from the intersection between the first field of view and the second field of view.

[0045] In one example, at least one sensing device may include a sensing device including an optical camera device and a rangefinder device. In other words, the optical camera device and the rangefinder device may be integrated into a single component, such as disposed in an enclosure. In another example, at least one sensing device may include two sensing devices, the first sensing device including the optical camera device and the second sensing device including the rangefinder device, wherein the first sensing device and the second sensing device may be physically separated from each other.

[0046] In another example, at least one sensing device may include multiple optical camera devices located at different positions (thus having different viewing angles), wherein each optical camera device may be configured to collect optical imaging data, and the multiple different viewing angles in the optical imaging data may provide ranging data. Exemplarily, at least one sensing device may include a stereo camera.

[0047] The sensing data is used to guide the movement of the gripping element. More specifically, the drive mechanism is configured to move the gripping element based on identification data obtained from the sensing data. In particular, the identification data is associated with the rack (i.e., associated with one or more empty receptacles and / or one or more sample containers).

[0048] Typically, obtaining identification data may include processing the sensed data, such as by processing the sensed data using object recognition techniques. In particular, processing the sensed data may include, for example, combining optical imaging data with range finding data when performing object recognition. Furthermore, additional data sets may be combined with the sensed data to obtain identification data, as discussed below.

[0049] As mentioned, at least one sensing device may be configured to collect sensing data associated with one segment of the stent at a time. In this case, for example, identification data may be obtained from the sensing data of each segment or may be obtained from sensing data associated with multiple segments, i.e., after at least one sensing device has accumulated sensing data associated with different segments.

[0050] The identification data may be obtained from the sensed data by a first computing device including at least one processor and a memory. The first computing device may receive the sensed data as input and provide the identification data as output. In a specific example, the first computing device may be included in the robotic manipulator. In another example, the first computing device may not be included in the robotic manipulator and may be in data communication with the robotic manipulator, for example, via a network such as an intranet or the Internet.

[0051] The drive mechanism is configured to move the gripping element based on the identification data. In particular, the identification data determines how the gripping element should be moved, in particular to which position the gripping element should be moved, for example for gripping a sample container from a rack or releasing a sample container into a rack.

[0052] Exemplarily, the drive mechanism can be configured to move the gripping element according to instructions generated based on the identification data. The instructions can also be based on one or more rules for manipulating the sample container, such as optimization rules for the overall route of the gripping element and priority rules about which sample containers should be manipulated first (e.g., due to the urgency of the analysis).

[0053] The instructions may include actual machine commands for the drive mechanism.The instructions may be generated by a computing device (which may be the first computing device or the second computing device) in communication with the drive mechanism.

[0054] Collecting both optical imaging data and range-finding data to acquire information about the rack results in accurate and precise detection of the actual, case-by-case environment that the robotic manipulator operates in. This enables the robotic manipulator to manipulate sample containers in a fine-tuned manner, thereby increasing throughput and minimizing the occurrence of errors.

[0055] Exemplarily, as mentioned, at least one sensing device may include an optical camera device and a rangefinder device, such as a LIDAR device. Using an optical camera device and a rangefinder device to collect optical imaging data and rangefinder data, respectively, allows simplifying the manufacture of the device, for example with respect to implementations in which rangefinder data is collected by multiple cameras via computer stereo vision. More specifically, even in the case of two separate devices, small errors in positioning the two sensing devices (i.e., in this case the optical camera device and the rangefinder device) relative to each other will not substantially affect the estimation of the distance included in the rangefinder data.

[0056] In a specific example, the identification data may include receiving portion occupancy information, which indicates whether at least one receiving portion of the sample container holder is empty or occupied by a sample container. In other words, the occupancy information indicates whether there are any sample containers in the receiving portion or whether there are no sample containers in the receiving portion (i.e., the receiving portion is empty). Consistent with the above explanation, the receiving portion occupancy information may only be related to a suitable subset of the plurality of receiving portions of the holder, or may be related to each of the plurality of receiving portions.

[0057] In particular, a sample container occupying a receiving portion of the rack is manipulated by the robotic manipulator while it is gripped / released by the gripping elements and / or while at least one sensing device collects sensory data related to said sample container.

[0058] It should be noted that the fact that the identification data and in particular the receiving portion occupancy information is obtained from sensing data including both optical imaging data and ranging data makes the receiving portion occupancy information more accurate. For example, from the optical imaging data alone (e.g., a digital image), a receiving portion may appear empty even if the receiving portion is occupied, for example because the occupied sample container has a dark cover and / or because the ambient lighting conditions are suboptimal (e.g., a dark environment). The use of ranging data in addition to the optical imaging data can correct possible false occupancy detections that would occur if only one set of data was used.

[0059] Since the receptacle occupancy information is part of the identification data, the drive mechanism may be configured to move the gripping element based on the receptacle occupancy information.In an illustrative example, the rack shall be unloaded by using a robotic manipulator.

[0060] In some cases, as mentioned above, the sensed data may only be related to sections of the support. In particular, the clamping element may be positioned relative to the support so that the sensed data collected by the at least one sensing device is related to a first section of the support. Therefore, the identification data may include information indicating whether the receiving portion in the first section of the support is occupied or empty, for example which receiving portions in the first section are occupied and which receiving portions are empty. If at least one receiving portion in the first section is occupied by a sample container, the drive mechanism may be instructed to move the clamping element toward the receiving portion to clamp the sample container in the receiving portion. If all receiving portions in the first section are empty, the drive mechanism may be instructed to move the clamping element to a second position relative to the support so that the sensed data collected by the at least one sensing device is related to a second section of the support.

[0061] In other cases, as mentioned above, the sensed data may be related to the entire rack. Thus, the identification data may include information indicating which receptacles of the rack are occupied and which receptacles of the rack are empty. The drive mechanism may be instructed to move the gripping elements only one at a time to the occupied receptacles of the rack to grip the corresponding sample container.

[0062] Thus, in all cases, the gripping elements can be guided dynamically, i.e. based on the actual arrangement of the sample container in the rack, to the occupied receptacles, without having to rely on predetermined movement rules which lead to suboptimal, time-consuming and energy-consuming movements of the gripping elements. This also enables an operator or a machine loading the rack, for example at an access station in a laboratory or at a doctor's office, to freely place the sample containers in the rack without having to follow specific rules.

[0063] As a result, the throughput of the robot manipulator can be effectively increased, in particular while also simplifying the preparation phase of the sample containers before they are manipulated by the robot manipulator.

[0064] In a specific example, in the case where the receiving portion is occupied by a sample container, the identification data also includes container position information indicating the position of the sample container. Additionally or alternatively, in the case where the receiving portion is empty, the identification data may also include receiving portion position information indicating the position of the empty receiving portion.

[0065] In particular, the receiving portion occupancy information may indicate whether two or more receiving portions of the sample container holder are empty or occupied, a receiving portion in a part of the two or more receiving portions is empty, and each receiving portion in another part of the two or more receiving portions is occupied by a corresponding container. In this case, the identification data may include both the container position information and the receiving portion position information, or only the container position information or only the receiving portion position information.

[0066] The container position information may indicate the corresponding positions of the plurality of sample containers in the rack, that is, the position of each sample container in the rack. The receiving portion position information may indicate the corresponding positions of the plurality of empty receiving portions of the rack, that is, the position of each empty receiving portion of the rack.

[0067] The position of the sample container and / or the position of the empty receiving portion can be a position in three-dimensional space. Therefore, the position of the sample container, in particular the position of the top portion of the sample container (e.g., the position of the center of the portion) can be given by three numerical values ​​(e.g., expressed in cm or mm). The top portion of the sample container can be the upper surface of the opening or cover of the sample container. Similarly, the position of the empty receiving portion, in particular the position of the upper opening of the empty receiving portion (e.g., the position of the center of the upper opening) can be given by three numerical values ​​(e.g., expressed in cm or mm). For example, the position of the sample container occupying the receiving portion of the support is the position of the receiving portion.

[0068] Exemplarily, the identification data may include information indicating the position of the empty receiving portion of the support and / or the position of the sample container in the receiving portion of the support. In particular, the position of the object (e.g., empty receiving portion or sample container) may be specified by a triplet value, which specifies the position of the object in a reference system and / or relative to at least one sensing device or clamping element. In particular, the triplet value may be Cartesian coordinates, spherical coordinates or cylindrical coordinates.

[0069] The position may be an absolute position, for example the value is a coordinate in a coordinate system relative to the origin of the coordinate system, or may be a relative position relative to another element, such as a reference point on a clamping element or a bracket.

[0070] The relative position of a given object (sample container or empty receiving portion) relative to another element can be given by three numerical values ​​indicating the components of the spacing between the given object and the other element along the three axes of the coordinate system. Therefore, the information specifying the relative position of the given object relative to the other element specifies the distance between the given object and the other element. Using the relative position of the given object relative to the other element and the absolute position of the other element, the absolute position of the given object can be obtained.

[0071] The drive mechanism can be configured to move the gripping element based on the container position information. Exemplarily, the drive mechanism can move the gripping element (e.g., from a starting position, such as an idle position or a previous working position) to the position of the sample container so as to grip the sample container from an occupied receiving portion.

[0072] The drive mechanism can be configured to move the gripping element based on the receiving portion position information. Exemplarily, the drive mechanism can move the gripping element (e.g., from a starting position, such as an idle position or a previous working position) to the position of an empty receiving portion so that the sample container is released in the empty receiving portion.

[0073] Therefore, the drive mechanism can move the gripping element directly to a position of interest according to the situation (e.g., loading or unloading), wherein the position of interest is accurately determined due to the combination of the optical imaging data and the distance measurement data, thereby avoiding meaningless movement of the gripping element. Therefore, the throughput of the robot manipulator can be improved.

[0074] In a specific example, where the receiving portion is occupied by a sample container, the identification data may include information indicating one or more characteristics of the sample container in addition to the container position information. The characteristics of the container may include the shape of the container, the type of container, one or more of the dimensions of the container (such as height or diameter), the presence of an internal liquid (in the case where the sample container is not capped).

[0075] The drive mechanism can be configured to move the gripping element based on information indicating one or more features of the sample container and the container position information. Exemplarily, the drive mechanism can move the gripping element to the position of the sample container, and / or the speed of the gripping element can be adjusted when lifting the sample container based on whether the sample container contains liquid.

[0076] Alternatively or additionally, the gripping element may be instructed, for example by the second computing device, to manipulate the sample container based on information indicating one or more characteristics of the sample container. For example, the strength with which the gripping element grips the sample container may depend on one or more characteristics of the sample container, such as one or more characteristics of the sample container type. For example, the position of the gripping element with which the gripping element grips the sample container may depend on one or more characteristics of the sample container, such as the container type and / or one or more dimensions of the sample container.

[0077] In a specific example, in the case where the receiving portion is occupied by a sample container, the identification data may also include closure information indicating whether the sample container is closed by a cover. In other words, for a given sample container held in the rack, the closure information may indicate whether the given sample container includes only a vessel or both a vessel and a cover.

[0078] The presence of a cap can be detected more accurately by obtaining identification data from both the optical imaging data and the ranging data. In one example, the rack can have been processed at the decapping station before being manipulated by the robotic manipulator. Thus, due to the at least one sensing device, a decapping quality check can be performed, i.e., it can be ensured that all caps are successfully removed at the decapping station.

[0079] Thus, for example, it may be expected that a sample container in a rack is uncapped. If, according to the closure information, the sample container is not uncapped as expected, the drive mechanism may move the gripping element away from the sample container and onto the uncapped sample container. In some examples, additionally, an error indicating an uncapped sample container may be reported, and the sample container may be transferred to a suspicious sample area for manual inspection by an operator. The same applies, mutatis mutandis, to the case where the sample container in the rack is expected to be capped.

[0080] In addition, in a specific example, in the case where the sample container is closed by a cover, the identification data may also include cover information indicating one or more features of the cover. The features of the cover may include one or more of the type of cover, the shape of the cover, the size of the cover (e.g., cover diameter, cover height) and / or the color of the cover. In particular, the features may be associated with a protruding portion of the cover. At least some features of the cover may be more accurately detected by obtaining identification data from both the optical imaging data and the ranging data. Features of the cover, such as the color of the cover, may indicate whether the tube contains a STAT sample.

[0081] The drive mechanism can be configured to move the clamping element based on the cover information. In an example, the receiving portion occupancy information can indicate that the first receiving portion is occupied by the first sample container and the second receiving portion is occupied by the second sample container. According to the cover information, the first sample container can have a cover indicating a common sample inside (e.g., via color), and the second sample container can have a cover indicating a priority sample inside (e.g., STAT sample). The drive mechanism can move the clamping element directly toward the second sample container because the processing of the second sample container is more urgent. Therefore, the manipulation of the sample container can be improved in terms of turnover.

[0082] As mentioned above, the first computing device can be configured to obtain identification data from the sensory data, and in a specific example, the first computing device can be included in the robotic manipulation device. Thus, at least one sensing device can be configured to provide the sensory data to the first computing device, and correspondingly, the first computing device can be configured to access the sensory data.

[0083] In the present disclosure, "accessing data by a device (e.g., a first computing device)" may include receiving data from another device (e.g., a sensing device) or retrieving data from a remote data storage (e.g., cloud storage, etc.). In particular, the computing device may retrieve the data in response to a notification sent by another computing device. Correspondingly, in the present disclosure, "providing data by a device" may include sending data to another device, transferring data to a remote data storage accessible by another device, or making data available to another device, for example, by sending a link to the data to the other device.

[0084] Exemplarily, at least one sensing device and the first computing device may be in data communication. Alternatively, at least one sensing device may include the first computing device.

[0085] The first computing device is configured to generate identification data using the sensory data, ie, create the identification data. In some examples, the first computing device may generate the identification data by using the sensory data and one or more other data sets.

[0086] Exemplarily, the first computing device may be configured to obtain identification data in two stages. In the first stage, the first computing device may use the sensed data to obtain intermediate data (the intermediate data may already include a portion of the identification data), and in the second stage, the first computing device may use the intermediate data combined with one or more other data sets to obtain identification data (the remainder of the identification data).

[0087] In the first stage, the first computing device can use object recognition technology. In particular, the intermediate data can include receiving portion occupancy information and optionally include closure information and lid information. Alternatively or additionally, the intermediate data can include two-dimensional position information about the sample container and / or the empty receiving portion. The two-dimensional position information can indicate the position of the object (that is, the sample container or the empty receiving portion) in the digital image, for example, the position specified by the reference system of the digital image, which is two-dimensional and defined by the edge of the digital image. Therefore, obtaining identification data can include mapping the two-dimensional position to the three-dimensional position in the physical environment. In the second stage, the first computing device can use one or more other data sets to perform the mapping, as explained below.

[0088] Exemplarily, the first computing device may be configured to use a machine learning algorithm (hereinafter also referred to as: "MLA") to obtain identification data from the sensed data. In particular, the MLA is trained to recognize the presence of sample containers in the rack and / or empty receptacles of the rack, and to estimate the positions of the sample containers in the rack and / or empty receptacles of the rack if applicable. For example, if the sensed data comprises a digital image, the MLA is trained to recognize the presence of sample containers and / or empty receptacles, and to estimate the positions of the sample containers and / or empty receptacles in the digital image if applicable. In particular, the first computing device may use a machine learning algorithm in the first stage. Exemplarily, the MLA is a neural network, such as a trained neural network. The neural network may be a feed-forward convolutional neural network.

[0089] In particular, the MLA is configured to generate an output by processing an input. The input to the machine learning algorithm may include optical imaging data and ranging data. For example, a digital image and a thermal map may be merged to obtain an input frame, wherein the input frame includes a plurality of pixels and each pixel may be associated with a value for one or more component intensities and a value for a distance measurement. Exemplarily, the input frame may be an RGBD frame, wherein each pixel is associated with a value for red, green, blue, and distance. The input frame may be an input to the machine learning algorithm.

[0090] In particular, the output of the MLA may include identification data. If the sensed data includes a digital image, the output of the machine learning algorithm may include information indicating one or more locations of the digital image at which an object was identified. Thus, the latter information encodes the two-dimensional location and optionally the size of the identified sample container or lid of the sample container. The output of the MLA may include information indicating the type of object identified by the MLA.

[0091] For example, for each identified object, the latter information may include information specifying the type of object, i.e. whether the identified object is an empty receptacle, an uncovered sample container, or a covered sample container with a given cover type. Thus, the information indicating the type of object identified by the MLA may encode the receptacle occupancy information as well as the closure information and the cover information. The output of the MLA may include information indicating a confidence level of the identification. In particular, for each identified object, the latter information may include information specifying the confidence level of the identification of the identified object.

[0092] For example, the output of the MLA may include an output frame. Exemplarily, the output frame includes one or more pixel regions of a digital image, each region including a plurality of pixels. At least some of the pixel regions may be associated with a plurality of values ​​encoding intermediate data (i.e., sensed data) obtained from an input frame by a machine learning algorithm. For example, the output frame may include pixel regions where an object has been identified and / or pixel regions where an object has not yet been identified. Pixel regions where an object has not yet been identified may not have values ​​associated therewith, while pixel regions where an object has been identified may have a set of values ​​associated therewith.

[0093] This group of values ​​can include one or more values ​​that uniquely specify the pixel area and the position of the pixel area in the digital image. For example, if the pixel area is circular, one or more values ​​include the position of the pixel at the center of the area in the image and the value of the radius of the area. For example, if the pixel area is rectangular, one or more values ​​include the position of the pixel at the center of the area in the image, the length of the longer side of the rectangle and the length of the shorter side of the rectangle. In particular, if the area is square, one or more values ​​include the position of the pixel at the center of the area in the image and the length of the side of the square. A group of values ​​associated with the pixel area can also include a value indicating the confidence level of the recognition and a value indicating the type of the object recognized.

[0094] The location of the pixel area in the digital image can be mapped by the first computing device in a second stage to the three-dimensional location of the object identified in the area within the physical environment. To perform the mapping, the first computing device can combine the intermediate data with one or more other groups of data, the one or more other groups of data including one or more of the following: ranging data, support data, support position data, clamp position data.

[0095] In particular, mapping the position of a pixel region in a digital image to the three-dimensional position of an object identified in the region can be performed using a mapping program. Exemplarily, the mapping can be performed using a pinhole camera model. The model provides a mathematical relationship between the coordinates of a point in the three-dimensional world and the coordinates representing the point in the image. Geometric distortions can be included in the pinhole camera model, such as disclosed in the article "A Flexible New Technique for Camera Calibration" written by Zhengyou Zhang and published in IEEE Transactions on Pattern Analysis and Machine Intelligence (Vol. 22, No. 11, November 2000, pp. 1330-1334). The article is incorporated herein by reference.

[0096] In particular, if the sensing device comprises a camera, in particular an optical camera, the mapping may comprise estimating the three-dimensional coordinates (x , m ) of the object depicted at the center of the pixel area in the camera frame. 1 x 2 x 3 ). Coordinate x 3 It can be estimated by using the range data and distance information collected by the sensing device and the orientation (ie angle) of the optical axis of the camera relative to the vertical axis. Alternatively or additionally, the coordinate x3 It can be estimated by using the position of the upper surface of the bracket relative to the camera. 1 and x 2 It can be estimated by using the following formula:

[0097]

[0098] Among them, (a 1 a 2 ) is the coordinate of the center of the pixel region in the image, (u 0 v 0 ) are the coordinates of the principal point, α and β are scale factors in the axis of the image's reference system, and γ is a parameter describing the skewness of the axis of the image's reference system. The parameter u 0 、v 0 , α, β, γ can be estimated, for example, by using the calibration procedure disclosed in the article "A Flexible New Technique for Camera Calibration" written by Zhengyou Zhang.

[0099] Furthermore, mapping may include calculating the three-dimensional coordinates (y) in a world reference system, such as the system used by the fixture to locate points in three-dimensional space. 1 y 2 y 3 ) to estimate the 3D position of the object depicted at the center of the pixel region. 1 y 2 y 3 ) and (x 1 x 2 x 3 ) are related to each other by using a rotation and translation that transforms the world reference frame and the camera reference frame into each other. The coordinates (y 1 y 2 y 3 ) can be used to generate a method for making the clamping element move in the direction indicated by the coordinate (y 1 y 2 y 3 ) instructions to move at a position described in a world reference frame.

[0100] In summary, the mapping procedure may include converting coordinates representing points in the image into three-dimensional coordinates of a camera-related reference frame, and converting the three-dimensional coordinates of the camera-related reference frame into three-dimensional coordinates of a reference frame related to the drive mechanism and the clamp.

[0101] In a particular example, the first computing device can be configured to obtain identification data by using rack data and rack position data, wherein the rack data includes information indicating the position of each receiving portion of the rack in the rack, such as the position relative to a reference point of the rack, and the rack position data includes information indicating the position and / or orientation of the sample container rack.

[0102] The rack data may also include information indicating the rack type and / or one or more dimensions of the sample container rack.The rack type may indicate the capacity of the rack (eg, the number of rows and columns in which the receptacles are arranged in the rack).

[0103] The stand data may be accessed by the first computing device, such as from another device, from a remote data store. In one example, the stand position data may be accessed by the first computing device, such as from another device, from a remote data store.

[0104] In another example, the stent position data can be obtained by using the sensed data. In particular, the stent position data can be obtained by using a machine learning algorithm to identify one or more reference points on the stent in the image and by using the pinhole camera model described above to map the position of the one or more reference points in three-dimensional space. The position of the reference points allows the position and orientation of the stent in three-dimensional space to be determined.

[0105] By using the position and orientation of the support in three-dimensional space and information indicating the position of each receiving portion of the support, the position of each receiving portion of the support in three-dimensional space can be estimated. The three-dimensional position of the receiving portion (as obtained by using the position and orientation of the support in three-dimensional space and information indicating the position of each receiving portion of the support) can be compared with the three-dimensional position of the object identified in the image obtained by using the mapping procedure described above.

[0106] This comparison allows to provide a further estimate of the position of the objects identified in the image. In particular, the further estimate of each of the objects is the three-dimensional position of the receiving portion closest to each of the objects. In this case, if the clamping element is to be moved towards one of the objects identified in the image, the clamping element is instructed to move towards the receiving portion closest to the object. In this way, the accuracy of the estimate of the three-dimensional position of the objects identified in the image is improved.

[0107] Exemplarily, the rack data and rack position data can also be used to perform a "sanity check" on the identification data. If the identified object is located at a three-dimensional position outside the volume of the rack, the identified object should be discarded because the object cannot actually be an empty receptacle or a sample container in a receptacle.

[0108] In a specific example, the robotic manipulator can be configured to read the rack data, the rack data being included in a label of the sample container rack. For example, the label can be an RFID tag or can be an optical label, such as an element with a specific color and / or shape. In the former case, the robotic manipulator can include an RFID reader. In the latter case, the optical label can be part of the optical imaging data, and the rack data can be read from the optical imaging data by identifying the optical label (e.g., by the first computing device).

[0109] In the case of a spatial offset between the gripping element and the at least one sensing device in the xy plane, the first computing device may also be configured to obtain the identification data by using gripper position data comprising information indicating the position of the gripping element. The position of the gripping element may be given by the drive mechanism.

[0110] In particular, the spatial offset can be determined by using a reference point with a known three-dimensional position. At least one sensing device (e.g., an optical camera device) can be placed vertically on top of the reference point, so that the xy position of the at least one sensing device is also known, because this position coincides with the position of the reference point. The spatial offset is then given by the difference between the xy position of the clamping element and the xy position of the at least one sensing device.

[0111] A second aspect of the invention relates to a first computing device configured to obtain identification data from sensed data as discussed above. In particular, the (first) computing device may be configured to: access sensed data associated with a sample container holder, the sensed data comprising optical imaging data and range finding data; and obtain identification data associated with the sample container holder by using the sensed data.

[0112] A corresponding computer-implemented method may include: accessing, by a first computing device, sensory data associated with a sample container holder, the sensory data including optical imaging data and ranging data; and obtaining, by the first computing device, identification data associated with the sample container holder by using the sensory data. In a specific example, obtaining the identification data may be performed by using a machine learning algorithm, as described above.

[0113] A third aspect of the present invention relates to a system. The system comprises the robotic manipulation device described above and a second computing device, wherein:

[0114] The first computing device is configured to provide identification data to the second computing device;

[0115] The second computing device is configured to generate instructions for moving the gripping element based on the identification data; and

[0116] The drive mechanism is configured to move the clamping element upon command.

[0117] The first computing device and the second computing device may be in data communication with each other, for example via a network such as an intranet or the Internet or via a bus.The first computing device may be included in the robotic manipulator and the second computing device may be located external to the robotic manipulator.

[0118] The first computing device may send the identification data to the second computing device, which may use the identification data to generate instructions for moving the gripping element, ie, instructions for controlling the drive mechanism. The instructions may include actual machine commands for the drive mechanism.

[0119] The robotic manipulator can be configured to receive instructions from a second computing device. In one example, the first computing device can receive the instructions and use the instructions to control the drive mechanism. In another example, the drive mechanism can receive the instructions and be directly controlled by the second computing device, wherein the first computing device can act as an edge device.

[0120] The drive mechanism follows the command to move the gripping element, for example towards the support, to pick up or release the sample container.

[0121] A fourth aspect of the present invention relates to a method for manipulating a sample container, wherein the sample container is arranged or is to be arranged in a sample container holder. The method comprises:

[0122] - collecting sensing data related to the sample container holder by at least one sensing device connected to the clamping element, the sensing data comprising optical imaging data and distance measurement data;

[0123] - moving the gripping element based on identification data associated with the sample container holder by a drive mechanism configured to move the gripping element, wherein the identification data is obtained from the sensed data.

[0124] The method may further comprise obtaining, by the first computing device, identification data from the sensed data, optionally by using a machine learning algorithm.

[0125] Prior to collecting the sensing data, the method may further include performing a calibration of the at least one sensing device. In particular, the calibration may include performing a calibration procedure. For example, if the at least one sensing device includes a camera, the method further includes calibrating the camera. In particular, the camera may be calibrated by using a conventional calibration procedure, such as the calibration procedure disclosed in the article “A Flexible New Technique for Camera Calibration” by Zhengyou Zhang.

[0126] Prior to collecting the sensing data, the method may further include positioning the gripping element and the sample container holder relative to each other via the drive mechanism such that at least a portion of the holder is located within the sensing region.

[0127] Exemplarily, the method according to the fourth aspect of the invention may be a method for unloading a rack. For example, the sensing data may be associated with a segment of the rack around a first position in the xy plane. In particular, the identification data may include receiving portion occupancy information indicating whether a receiving portion in a segment of the rack is empty or occupied by a sample container and receiving portion position information or container position information, respectively. For example, the method may include:

[0128] - moving the clamping element by means of a drive mechanism so that at least one sensing device moves to a first position above the support;

[0129] - obtaining identification data by means of a first computing device;

[0130] If the receiving part is empty, then:

[0131] - moving the clamping element to a second position different from the first position by means of a drive mechanism;

[0132] And, if the receptacle is occupied by a sample container, then:

[0133] - moving the clamping element to the position of the sample container by means of a drive mechanism;

[0134] - Unloading the sample container from the rack by gripping the sample container with the gripping element.

[0135] In a specific example, the method according to the fourth aspect of the invention may be a method for loading a stent. For example, the sensing data may be associated with a segment of the stent around a first position in the xy plane. In particular, the identification data may include receiving portion occupancy information indicating whether a receiving portion in the segment of the stent is empty or occupied and receiving portion position information. For example, the method may include:

[0136] - moving the clamping element by means of a drive mechanism such that at least one sensing device moves to a first position above the support, wherein the clamping element holds the sample container;

[0137] - obtaining identification data by means of a first computing device;

[0138] If the receiving area is occupied, then:

[0139] - moving the clamping element to a second position different from the first position by means of a drive mechanism;

[0140] And, if the receiving section is empty, then:

[0141] - moving the clamping element along the z-axis to the position of the empty receiving portion by means of a drive mechanism;

[0142] The sample containers are loaded into the rack by releasing them in the empty receptacles by means of gripping elements.

[0143] In particular, the method may comprise gripping the sample container by the gripping element, for example before moving the gripping element to the first position above the support. BRIEF DESCRIPTION OF THE DRAWINGS

[0144] The details of the exemplary embodiments are described below with reference to the exemplary drawings. Other features will become apparent from the description, drawings and claims. However, it should be understood that although the embodiments are described separately, individual features of different embodiments may also be combined into other embodiments.

[0145] Figure 1 A schematic diagram of a system including a robotic manipulator is shown.

[0146] Figure 2 An exemplary schematic diagram of a sample container holder holding a sample container as identified by a robotic manipulator is shown.

[0147] Figure 3 A flow chart including the steps of an exemplary method for manipulating a sample container using a robotic manipulator is shown.

[0148] Figure 4 A swim lane diagram including exemplary steps performed by a robotic manipulator and a computing device is shown. DETAILED DESCRIPTION

[0149] In the following, a detailed description of the examples will be given with reference to the accompanying drawings. It should be understood that various modifications may be made to the examples. Unless otherwise expressly indicated, elements of one example may be combined and used in other examples to form new examples.

[0150] Figure 1 A schematic diagram of a system including a robotic manipulator 100 is shown. The robotic manipulator 100 includes a gripping element 120 and a drive mechanism 130 configured to move the gripping element 120 along a vertical axis A2 and a first horizontal axis A1 and a second horizontal axis (not shown), the second horizontal axis being orthogonal to the vertical axis A2 and the first horizontal axis A1. When brought into a suitable position by the drive mechanism 130, the gripping element 120 can take a pipe out of and put a pipe into a pipe support 400 positioned below the gripping element 120. Specifically, the gripping element 120 includes gripping fingers 121, 122 for gripping and releasing the pipe.

[0151] The support 400 rests on a holding element 150, such as the bottom of a drawer or a tabletop of a laboratory instrument. The support 400 is a C-shaped bracket having an upper plate and two lateral plates supporting the upper plate. The upper plate 470 of the support 400 includes a plurality of upper openings or receiving portions configured to receive and hold corresponding tubes. Figure 1 The bracket 400 may include twelve receiving portions arranged in two rows and six columns, wherein Figure 1 Only one row is shown.

[0152] For example, Figure 1 , tubes 410 to 450 are arranged in the first row of brackets 400, and receiving portion 460 is empty. More generally, bracket 400 can be in any configuration between full (i.e., all receiving portions are occupied by tubes) and empty (i.e., all receiving portions are empty). The tubes 410 to 450 arranged in bracket 400 can have different characteristics. For example, the tubes have different lengths relative to each other. In addition, tubes 430 and 450 are not capped, while tubes 410, 420 and 440 are capped, wherein the caps of tubes 410, 420 and 440 are different from each other.

[0153] The robotic manipulator 100 further comprises an optical camera device and a LIDAR device integrated in one sensing device 140. The sensing device 140 collects sensing data related to the support 400, specifically sensing data related to an empty receiving portion of the support 400 and sensing data related to an occupied receiving portion of the support 400, i.e., a tube occupying the receiving portion of the support 400. In particular, the sensing device 140 is positioned above the support 400 so that the optical camera device acquires a digital image of the support 400 from above and the LIDAR device measures the distance from the position of the LIDAR device to the upper plate of the support 400 and the upper parts of the tubes 410 to 450 protruding from the upper plate of the support 400.

[0154] According to the sensing characteristics of the sensing device 140, the sensing device 140 can collect sensing data about the entire support 400 once or step by step by scanning the support 400. The driving mechanism 130 can move the sensing device 140 to multiple positions, wherein the sensing device 140 collects sensing data about different sections of the support 400 at each of the multiple positions.

[0155] The robot manipulation device 100 also includes a first computing device 110, which includes a first processor (e.g., CPU, GPU, etc.) and a first memory. The first computing device 110 also includes a first network interface controller (NIC) configured to connect the first computing device 110 to one or more networks (e.g., intranet, Internet, cellular network, etc.). The first computing device 110 can communicate data with the sensing device 140, receive sensing data from the sensing device 140, and obtain identification data from the sensing device 140. Alternatively, the first computing device 110 can be a part of the sensing device 140.

[0156] The robot manipulator 100, and in particular the first computing device 110 of the robot manipulator 100, exchanges data with a second computing device 210. The second computing device 210 includes a second processor 211 (e.g., CPU, GPU, etc.) and a second memory 212. The second memory 212 may include a main memory and a secondary memory (not shown). The second computing device 210 includes an input / output (I / O) interface 210 for communicating with an input / output unit, for example, in the form of a screen. The second computing device 210 also includes a second NIC 214 configured to connect the second computing device 210 to one or more networks (e.g., an intranet, the Internet, a cellular network, etc.).

[0157] The first computing device 110 and the second computing device 210 may exchange data with each other via the first NIC 114 and the second NIC 214 by using a protocol suite such as TCP or IP. Figure 1 In particular, in some cases (see below for Figure 4 ), the second computing device 210 can generate instructions for controlling the drive mechanism 130 based on the identification data obtained by the first computing device 110.

[0158] Figure 1 The robot manipulator 100 is based on Figure 3 The method shown in Figure 4 In particular, the robotic manipulator 100 can unload tubes from at least partially occupied racks and can load tubes into at least partially empty racks.

[0159] Figure 3 The method 610 includes collecting sensing data at step 611. The sensing data is collected by the sensing device 140 and includes one or more color digital images (or RGB frames) of the support 400 acquired from above the support and one or more range heat maps (or LIDAR frames) of the support 400 as viewed from above.

[0160] At step 612, the first computing device 110 obtains recognition data by using the sensing data. In particular, the first computing device 110 may use a machine learning algorithm to generate recognition data from the sensing data. The machine learning algorithm may provide recognition functions for objects in the RGB frame and the LIDAR frame. The machine learning algorithm may include at least one neural network, such as a feedforward neural network. Exemplarily, the neural network may be a "You Only Look Once" (YOLO) object recognition neural network, in particular, version 3 of the YOLO object recognition neural network, as described in the paper "YOLOv3: An Incremental Improvement" written by J. Redmon and A. Farhadi and published on arXiv as arXiv:1804.02767v1. The paper is incorporated herein by reference.

[0161] In one example, the input of the neural network may be an RGBD frame, where each pixel is associated with a value for red, green, blue, and distance. The RGBD frame is obtained by merging the RGB frame and the LIDAR frame, thereby obtaining an input including four channels (e.g., channels "R", "G", "B" and a channel including information provided by the LIDAR). For example, accordingly, the number of channels in the input for the YOLO object recognition neural network may be set to four.

[0162] Optionally, before merging, one or more of the following pre-processing steps may be performed:

[0163] 1. Perform time synchronization for frames.

[0164] This step may be performed, for example, if the optical camera device and the LIDAR device continuously acquire pictures at different rates (e.g., 30fps vs. 25fps) or at the same rate but not in a synchronized manner. Temporal synchronization may be achieved by means of interpolation, such as by acquiring a frame from the camera device and by acquiring the average of the frames immediately before and after the camera frame from the LIDAR device. Alternatively, temporal synchronization may be achieved by approximation, i.e., by acquiring a frame from the camera device and by acquiring the closest frame in time from the LIDAR device.

[0165] 2. Align the RBG frame and the LIDAR frame to each other by using the rotations and translations that connect the camera reference frame and the LIDAR reference frame. These rotations and translations can be obtained by using a conventional LIDAR camera calibration procedure.

[0166] 3. Adjust the scaling of LIDAR frames and RBG frames.

[0167] For example, this step may be performed if the neural network requires a specific number of pixels and / or a specific size of file for input. Exemplarily, the LIDAR frames and RGB frames may be scaled to have 416×416 pixels.

[0168] 4. Perform normalization for LIDAR frames and RGB frames.

[0169] For example, assuming that distance is represented by an unsigned 16-bit integer, the values ​​in the RGB frame can be divided by 256, while the values ​​for the D channel can be divided by 65536, so that the image pixel values ​​for the LIDAR frame and the RGB frame can be scaled to values ​​between 0 and 1.

[0170] The neural network can output an output frame comprising a plurality of pixels associated with regions and confidence values ​​for the presence or absence of tubes having certain characteristics within these regions. Pixel regions are associated with a set of six numerical values, for example (0.95, 0.5, 0.5, 0.2, 0.2, 0). The value 0.95 indicates the confidence level, (0.5, 0.5) is the normalized object position, (0.2, 0.2) is the normalized object size, and 0 indicates an empty receptacle. A value of 1 for the latter value indicates a sample container with a first type of lid, a value of 2 indicates a sample container with a second type of lid, and a value of 3 indicates a sample container with a third type of lid. The type of lid can be distinguished by one or more of the following features: lid type, lid shape, lid size, lid color (particularly the color and size of the top ring). Finally, a value of 4 for the sixth numerical value can indicate an uncapped tube. Thus, for example, the number of categories of objects recognized in the output of the YOLO object recognition neural network can be set to five (empty receptacles, capped tubes with first type of caps, capped tubes with second type of caps, capped tubes with third type of caps, uncapped tubes).

[0171] The training data for the neural network includes pairs of input frames and output frames. In particular, the training data may include sensory data, i.e., images and distance heat maps, and outputs, such as output frames, that the MLA will generate by processing the sensory data. For example, the output frame includes pixel areas where objects have been identified and / or pixel areas where objects have not yet been identified. As mentioned above, pixel areas where objects have not yet been identified may not have values ​​associated with them, while pixel areas where objects have been identified may have a set of values ​​associated with them. The training data may be manually or automatically labeled by using a robotic arm configured to attempt to grip each position and making pictures to be labeled based on the results of the robotic arm's attempts.

[0172] For the exemplary case where the MLA is a YOLO object recognition neural network, the training process may, for example, include labeling about 200 input frames from different types of tubes and stents, wherein approximately 10 to 40 objects are labeled within each frame. The labeled data may be split into two groups according to a ratio of 80 / 20, i.e., 80% for the training phase and 20% for the validation phase. Training may be performed for 300 epochs with the batch size set to 2. A standard loss function defined for YOLO v3 as described in "YOLOv3: An Incremental Improvement" by J. Redmon and A. Farhadi, arXiv:1804.02767v1, is used.

[0173] Figure 2 An exemplary schematic output frame is shown in . In particular, the output frame has been generated by the neural network by processing the RGB frame acquired by the optical camera device 140 and the LIDAR frame acquired by the LIDAR 140. The output frame 500 includes a visual representation of pixel regions 530a to 530d, 540a to 540d, 550, 560, and 570a to 570b in which objects (e.g., tubes and empty receptacles) have been identified. The output frame 500 also includes a depiction 520 of the upper plate 470 of the support 400 and a depiction 510 of the retaining element 150 on which the support 400 is retained. In addition, it is recognized that regions 540a to 540d depict empty receptacles 460 of rack 400, regions 530a to 530d depict receptacles occupied by uncapped tubes 430, 450, region 550 depicts receptacles occupied by tubes 440 capped with a first type of cap, region 560 depicts receptacles occupied by tubes 410 capped with a second type of cap, and regions 570a to 570b depict receptacles occupied by tubes 420 capped with a third type of cap. Exemplarily, the second type of cap may indicate a STAT sample, i.e., a sample that should be processed with the highest priority. In other words, the second type of cap indicates that tube 410 contains a STAT sample.

[0174] Step 612 includes mapping the positions of pixel regions 530a to 530d, 540a to 540d, 550, 560, and 570a to 570b in the digital image to the three-dimensional positions of objects identified in the regions. In particular, the mapping can be performed using the mapping program described above. The identification data includes information indicating the three-dimensional positions of objects identified in pixel regions 530a to 530d, 540a to 540d, 550, 560, and 570a to 570b.

[0175] After the identification data has been obtained, the first computing device 110 obtains instructions for controlling the drive mechanism 130 by using the identification data (step 613). Figure 2 For example, if the robotic manipulator 100 must unload the bracket 400, the first computing device 110 may generate the following instructions, which instruct the drive mechanism 130:

[0176] - moving the gripping element 120 to the three-dimensional position of the tube 410 depicted in the area 560 to grip the tube 410 (because the tube 410 contains a STAT sample);

[0177] - moving the clamping element 120 to the unloading position to release the tube 410;

[0178] - Moving the gripping element 120 to the three-dimensional position of the tube 420 depicted in area 570a to grip the sample tube 420 , which three-dimensional position is adjacent to the three-dimensional position previously depicted in area 560 .

[0179] The instructions may also instruct the drive mechanism to move the gripping element 120 to all positions where the sample container is identified. Although this illustrative example is for unloading tubes, the same principles apply mutatis mutandis to loading.

[0180] Finally, the method 610 includes moving the clamping element 120 in response to the command by the drive mechanism 130 (step 614 ).

[0181] Figure 3 The method may be performed in a slightly different manner involving interaction between the robotic manipulator 100 and the second computing device 200, such as Figure 4 . Steps 611 and 612 are identical. Once the first computing device 110 has obtained the identification data, the first computing device 110 provides (eg, sends) the identification data to the second computing device 200 at step 621. Thus, at step 613', it is the second computing device 200, rather than the second computing device 200, that is responsible for the identification data. Figure 3 The first computing device 110 in step 613 obtains instructions for controlling the drive mechanism 130 by using the identification data. The instructions obtained by the second computing device 200 at step 613′ are the same as those obtained at step 614′. Figure 3 The same instructions are generated at step 613 of the flowchart shown in and described above.

[0182] Subsequently, at step 622, second computing device 200 provides (e.g., sends) instructions to robotic manipulator 100. In one example, second computing device 200 may control drive mechanism 130 and thus provide instructions directly to drive mechanism 130. In another example, second computing device 200 may provide instructions to first computing device 110 of robotic manipulator 100, and first computing device 110 may control drive mechanism 130.

[0183] The final step 614 of moving the gripping element 120 by the drive mechanism 130 in accordance with the command is also related to Figure 3 The same as step 614 shown in .

Claims

1. A robotic manipulator for manipulating a sample container, in, The sample container is arranged or to be arranged in a sample container holder, and the robot manipulation device comprises: - a gripping element for gripping and releasing the sample container; - a drive mechanism configured to move the clamping element; and - at least one sensing device connected to the holding element, wherein the at least one sensing device is configured to collect sensing data related to the sample container holder, the sensing data comprising optical imaging data and ranging data; Wherein, the drive mechanism is configured to move the gripping element based on identification data associated with the sample container holder, the identification data being obtained from the sensing data.

2. The robot manipulator according to claim 1, in: The at least one sensing device includes an optical camera device and a rangefinder device.

3. The robot manipulator according to claim 1 or 2, in, The sample container support comprises a plurality of receiving portions. And wherein the identification data comprises receptacle occupancy information indicating whether at least one receptacle of the sample container holder is empty or occupied by a sample container.

4. The robot manipulator according to claim 3, in, In the case where the receiving portion is occupied by the sample container, the identification data further includes container position information indicating the position of the sample container.

5. The robot manipulator according to claim 3 or 4, in, In the case where the receiving portion is empty, the identification data further includes receiving portion position information indicating the position of the empty receiving portion.

6. The robotic manipulator according to any one of claims 3 to 5, in, In case the receiving portion is occupied by the sample container, the identification data further comprises closure information indicating whether the sample container is closed by a cover.

7. The robot manipulator according to claim 6, in, In case the sample container is closed by a cap, the identification data further comprises cap information indicating one or more characteristics of the cap.

8. The robotic manipulator according to any one of the preceding claims, further comprising a first computing device, in, The first computing device is configured to obtain the identification data from the sensed data, and wherein, optionally, the first computing device is configured to use a machine learning algorithm to obtain the identification data.

9. The robot manipulator according to claim 8, in, The first computing device is configured to obtain the identification data by using: rack data, the rack data comprising information indicating a position of each receiving portion of the sample container rack in the sample container rack; and The rack position data includes information indicating the position and / or orientation of the sample container rack.

10. The robot manipulator according to claim 9, in, The robotic manipulator is further configured to read the rack data, the rack data being included in the indicia of the sample container rack.

11. The robot manipulator according to claim 8 or 9, in, The first computing device is configured to obtain the support position data by using the sensing data.

12. A robotic manipulator according to any one of claims 8 to 11, in, The first computing device is configured to obtain the identification data by using gripper position data comprising information indicative of a position of the gripping element.

13. A system, include: A second computing device and a robotic manipulation device according to any one of claims 8 to 12, wherein: the first computing device being configured to provide the identification data to the second computing device; The second computing device is configured to generate instructions for moving the gripping element based on the identification data; and The drive mechanism is configured to move the clamping element according to the command.

14. A method for manipulating a sample container, in, The sample container is arranged or is to be arranged in a sample container holder, and the method comprises: - collecting sensing data related to the sample container holder by at least one sensing device connected to the clamping element, the sensing data comprising optical imaging data and distance measurement data; - moving the gripping element by a drive mechanism configured to move the gripping element based on identification data associated with the sample container holder, wherein the identification data is obtained from the sensed data.

15. The method of claim 14, further comprising obtaining, by a first computing device, the identification data from the sensory data.

Images