SYSTEM FOR AUTOMATIC DETECTION OF LABORATORY WORK ITEMS AND METHOD FOR OPERATION OF A SYSTEM FOR AUTOMATIC DETECTION OF LABORATORY WORK ITEMS

Description

[0001] The present invention relates to a method for operating a system for the automatic recognition of laboratory work items according to claim 1. Furthermore, the invention relates to a system for the automatic recognition of laboratory work items according to claim 5.

[0002] In modern medical and scientific laboratories, various tasks, such as more or less complex laboratory routines, are performed partly manually, partly automatically, and partly through a combination of manual and automated processes. Even with laboratory equipment, such as laboratory robots, which can largely perform or process complex laboratory routines autonomously, it is still necessary for the manual loading of the laboratory robot by a user. Consequently, the execution of such laboratory routines remains prone to errors, even with the increasing automation. This, in turn, leads to missing or inaccurate results, or at the very least, to increased time and costs.

[0003] To reduce the error susceptibility of manual and / or automated steps in laboratory routines, methods and systems are already known in the art for optically monitoring the preparation, execution, and follow-up of a laboratory routine performed by a laboratory facility. This optical monitoring generally serves to prevent errors and malfunctions. Systems and methods are known in the art for this purpose in which optical specifications, for example, in the form of a reference image or reference representation, exist for a corresponding laboratory routine or a part or individual step of a laboratory routine. These specifications are compared with an actual image or representation obtained during the current execution of the laboratory routine to determine whether the laboratory routine is being performed correctly.Furthermore, methods and systems are already known in which structures, such as contours or edges, are recognized in an actual image using classical image processing in order to compare the recognized structures with predefined structures or reference structures and thereby obtain an indication of the correct and complete execution or processing of a laboratory routine.

[0004] The devices and methods of the prior art have the disadvantage of being generally very inflexible and overly dependent on some kind of fixed specification, such as a reference image. This leads to the problem that, with the known methods and devices, even different lighting conditions make comparing or aligning the actual image with a specification or reference difficult or impossible. Furthermore, the known methods and devices are not capable of flexibly and continuously or consistently monitoring a laboratory routine without a fixed trigger or initialization mechanism linked to a specific point in the routine. This would prevent them from acquiring and potentially further processing important information regarding the execution or completion of the laboratory routine.

[0005] Against this background, the present invention aims to propose an improved method for operating a system for the automatic detection of laboratory equipment, enabling flexible use and, in particular, enabling successful operation largely independent of lighting conditions and other external or environmental influences. Document EP2013 / 113929 discloses a system and method for tracking surgical instruments in an operating room. The system uses cameras to detect instruments in predefined areas and to store their positions.

[0006] This problem is solved with respect to the method by the features of claim 1 and with respect to the system by the features of claim 5.

[0007] Advantageous embodiments of the invention are specified in the dependent claims. The scope of the invention encompasses all combinations of at least two claims appearing in the description and / or features disclosed in the figures.

[0008] The task is accordingly solved by a method for operating a system (1) for the automatic recognition of laboratory work items and their condition in the area of ​​a plurality of designated storage positions (2) of a laboratory work area (3) comprising the following steps: - generating a single two-dimensional image of a laboratory work area (3) with an imaging unit (5) for generating two-dimensional images, wherein the work area (3) comprises a matrix-shaped arrangement of designated storage positions (2); - identifying at least one first evaluation area in the two-dimensional image with an identification unit (9) of the system (1), wherein the at least one evaluation area is arranged and mapped in the area of ​​a designated storage position (2) of the work area (3);- Processing the image, with a processing unit (10) of the system (1), into a classification pre-stage with regard to cropping to the image of the evaluation area and / or formatting the image of the evaluation area; - Automatic object recognition based on the classification pre-stage by applying at least one classification procedure based on feature generation and subsequent assignment to an object class with the features generated by the feature generation using a classification unit (11) of the system (1); - Generation of an output signal identifying the recognized object by means of an output unit; - Transmission of the output signal to a laboratory facility (4) of the system (1), with a communication unit (8) of the system (1), and consideration of the recognized object in the execution of a laboratory routine of the laboratory facility (4), namely the control of a robot arm;wherein feature generation is based on manually generated and / or machine-learned selection criteria by artificial neural networks and / or the assignment of the generated features to an object class is based on artificial neural networks; wherein the at least one classification procedure comprises at least a multi-stage classification procedure based on a convolution of the image contents of the classification pre-stage taking place at least in one stage with a set of operators trained for object recognition, namely convolution operators, with a neural network unit (111);wherein, when applying the multi-stage classification procedure, an object is initially recognized in the first stages of the artificial neural network according to a first coarse object class, and with progressive stages, finer structures are recognized by the respective stage, and subsequently, with each subsequent classification procedure carried out by the classification unit (11), correspondingly cascaded or nested subclasses of the object are recognized; wherein the representation includes color information in individual color channels, and preferably brightness information in a brightness channel, wherein, within the framework of the automatic object recognition, at least the color information of the individual channels is subjected to feature generation, wherein the channels are examined individually and the results of the examination are combined for feature generation;wherein the subclasses are defined in such detail by the trained operators that, in addition to the object itself, an object state is also recognized; wherein each subsequent classification procedure processes a cascaded or nested partial evaluation area of ​​the at least one first evaluation area; wherein, in a first stage of the multi-stage classification procedure, an object class tip holder is recognized; wherein, by selecting one or more partial evaluation areas in the multi-stage classification procedure, the classification unit recognizes whether and at which positions of the tip holder pipetting tips are arranged; and wherein, in the multi-stage classification procedure, the classification unit (11) further determines in which existing pipetting tips a liquid is located, which liquid and at what fill level.

[0009] The basic idea of ​​the invention is therefore based on the use of methods for the automatic recognition of properties (features) and the assignment of these properties or features to an object class within an object classification, whereby the automatic recognition is achieved through the use of pre-provided intelligent operators for feature generation. This enables flexible recognition of laboratory objects and their states at any given time, independent of a static reference or target.This means that, to a particularly advantageous extent, for all laboratory items or laboratory work items for which a corresponding set of trained operators for application in the classification procedure of the classification unit of the system has been generated or is available, corresponding item recognition can be carried out at any time within the scope of the laboratory routine and taken into account during the processing of the laboratory routine, regardless of lighting conditions or other environmental or external influences.

[0010] This makes the proposed method highly flexible and largely autonomous, meaning that no reference images or other optical target states need to be defined, but only that a prior training or provision of a corresponding trained set of operators (kernel) needs to be ensured in order to recognize the corresponding laboratory objects in every situation of a laboratory routine under a wide variety of environmental, especially lighting, conditions.

[0011] It has proven particularly advantageous that the region optically captured by the imaging unit (field of view) is directed towards and maps a laboratory work area by arranging target storage positions for corresponding laboratory work items. In laboratory automation systems, these target storage positions within a laboratory work area can be marked by optical and / or mechanical means and generally serve to enable or at least facilitate the positioning of a robot arm or other device of a laboratory automation system or comparable laboratory equipment within the target storage area.

[0012] The inventive automatic object recognition using a classification method based on feature generation and subsequent assignment to an object class with the features generated by the feature generation makes it particularly advantageous to ensure that the imaging unit has relatively low requirements for the quality of the optical image. Thus, reliable object recognition is possible with the proposed method even when only imaging units from the consumer segment, such as the "Logitech C930e" camera, are used.The laboratory equipment or tools that can be detected or are intended to be detected by this method can, in principle, be any type of object that is regularly or routinely used in relevant laboratory work or procedures. Preferably, these are laboratory tools used in the handling of liquid samples during relevant laboratory routines by well-known laboratory equipment, such as liquid handling robots. The following are just a few examples of items or laboratory equipment that can be detected dynamically and independently of the environment by this method: tip holders, tip boxes, thermoblocks, thermoadapters, various microtiter plates, and many more.

[0013] The identification of the evaluation area can be achieved in several different ways using the proposed method. Various possibilities and related details will be discussed further below. The image processing for separating or isolating the evaluation area within the image employs familiar image transformations, which can be performed using a corresponding digital transformation unit within the system's processing unit.

[0014] The classification process consists of feature generation and subsequent classification assignment based on these identified and / or generated features. Feature generation can preferably be based on manually created or machine-learned selection criteria, particularly through artificial neural networks.

[0015] Classification methods can employ techniques such as Nearest Neighbors, Linear SVM, RBF SVM, Gaussian Process, Decision Tree, Random Forest, Neural Net, AdaBoost, Naive Bayes, QDA, machine learning classifiers, and artificial neural networks. Generally, classification methods can be structured to first identify coarser structures in the representation, particularly in the preliminary classification stage, and then extract, recognize, and / or generate progressively finer structures or features.

[0016] The result of carrying out at least one classification procedure, with a suitably trained set of operators, is a highly reliable recognition of objects, especially laboratory equipment.

[0017] The method according to the invention thus enables the system to detect the laboratory work area, in particular a plurality of target storage positions within a laboratory work area, with particularly high reliability, either at specific points or continuously during any laboratory routine. This allows the system to derive useful information for carrying out or processing the laboratory routine from the detected information, especially from the detected objects and, if applicable, their states. Accordingly, the execution or processing of the laboratory routine can be dynamically or individually adapted to a detected situation and the corresponding detected objects in the target storage areas of the work area.

[0018] Such dynamic situation detection on a laboratory setup is by no means possible with existing static, optical control methods and systems. For example, the proposed method could detect that a microtiter plate and a tip holder are positioned in reversed locations on the laboratory work area, compared to a predefined or predetermined position for the automated execution of a laboratory routine.The inventive method allows the predefined or predetermined laboratory routine to be carried out safely, reliably and without errors by generating an output signal and transmitting and considering the information contained in the output signal regarding the detected objects, since despite the deviation from an existing specification, the deviation from the specification, namely the differently arranged laboratory work objects, can be reliably detected and the execution of the laboratory routine, for example the control of a robot arm, can be adjusted accordingly.

[0019] According to an advantageous embodiment of the method, it may be provided that the at least one classification procedure comprises at least a single-stage or multi-stage classification procedure based on a convolution of the image contents of the classification pre-stage taking place at least in one stage with a set of operators trained for object recognition, in particular convolution operators, with a neural network unit.

[0020] Multi-stage classification methods, i.e., artificial neural networks, can involve multi-stage classification processes in which the neurons of a stage or layer are connected to all neurons of a preceding or higher-level stage or layer. Various types of matrices can be used as operators, also called kernels, and these can be configured differently depending on the task of the respective layer or kernel. In principle, however, the classification methods are structured so that the first stages or layers identify coarser structures in the representation, particularly in the preliminary classification stage. With each subsequent stage or layer, finer structures or features are recognized or identified, ultimately enabling reliable object recognition.It may be stipulated that the image content of the classification pre-stage is convolved with all operators, particularly kernel operators, at each stage. This is also referred to as generating a feature map from the image data, which becomes increasingly dimensional with an increasing number of stages or layers of the neural network or classification process. In addition to the layers or stages where convolution with the operators or kernels is performed, the embedding of intermediate stages or layers may also be provided, enabling intermediate processing or preparation. For example, pooling layers and / or drop-out layers may be included.

[0021] A particular advantage of a multi-stage classification procedure is that the individual classifiers or operators can be more specialized and therefore both more precise and computationally more cost-effective.

[0022] Another preferred embodiment of the method involves transforming the generated two-dimensional image by a transformation unit, taking into account the imaging properties of the transformation unit, in particular by perspective manipulation. This can advantageously facilitate and accelerate the subsequent image processing steps and the classification procedure, and / or reduce the variance of the image sections.

[0023] In the case of an unknown position and / or orientation of the imaging unit, or if the evaluation areas cannot be selected with minimal overlap, a further particularly advantageous embodiment of the method may provide for the use of a referencing unit that can detect changes over time in images generated by the imaging unit.

[0024] To determine at least one evaluation area, a comparison of the image with a reference image is performed. In order to operate or utilize the hardware as efficiently as possible, the referencing unit can also be used to perform a classification for evaluation areas, in particular a comparison with a reference image, only if a change or image alteration between a current image and the reference image is detected that lies above a threshold value defined as an area.

[0025] This means that, first, a comparison is carried out for predetermined sub-areas of the image to determine whether image content other than just the workspace is depicted in the respective sub-area.

[0026] This design offers several advantages. Firstly, identifying the evaluation area by comparing it to the reference unit is a classic image processing approach that can draw on a variety of known methods, algorithms, and procedures. However, to ensure that a complete image of a laboratory object to be recognized is included in the evaluation area—which forms the basis for reliable automatic object recognition using the classification method—it is particularly advantageous to verify, for any sub-area where it is determined that image content other than the laboratory work area is depicted, whether the image content representing the corresponding laboratory object is completely encompassed by the image area or extends beyond it.Accordingly, an adjustment of the boundaries of the sub-area may also become sensible or necessary.

[0027] According to an advantageous embodiment, the identification of the evaluation area can therefore be realized by identifying the background and the foreground that stands out from this background, wherein the background, which is depicted by the reference image, represents in the broadest sense the statistically most frequently occurring image values ​​within a time interval, whereas the foreground accordingly necessarily depicts objects placed or removed in the laboratory work area, in particular in the intended storage positions of the laboratory work area, which thus form the candidates for automatic object recognition with the classification method.The sub-areas, each of which is individually used as the basis for or compared to corresponding reference images or sub-areas of a reference image in the identification document for the identification of an evaluation area, can preferably subdivide the laboratory work area into segments. Particularly preferably, for the described determination of an evaluation area, a color evaluation of the images or the sub-areas of the image with reference image, in particular corresponding sub-areas of the reference image, can be carried out using a referencing unit encompassed by the identification unit. The brightness information, insofar as it is available, can also be compared with each other in the respective sub-areas of the image and the reference image.

[0028] According to a further particularly advantageous embodiment of the present method, it can be provided that, following automatic object recognition or object recognition, a detail recognition unit performs a geometric evaluation, in particular an automatic recognition of basic geometric structures, preferably because this allows additional information to be determined independently of an object class, thus enabling not only pure classification but also regression. This regression can be performed using classical image processing or machine learning (e.g., support vector machines or neural networks).

[0029] This can be used, for example, to detect the tilting of a laboratory work item or a laboratory work item that is incorrectly or twistedly positioned.

[0030] Furthermore, within the framework of automatic object recognition, a tip holder or point holder arranged in an evaluation area of ​​an image can be detected. In a subsequent detailed recognition process or step, the detailed recognition unit can use known recognitions or contour recognitions, via classical image analysis or evaluation, to determine in which images or positions of the tip holder or point holder points are located. Other detailed recognitions of objects detected by the classification unit, especially those detected automatically, can also be achieved advantageously and in a particularly resource-efficient manner for the system via the described automatic recognition of basic geometric structures.

[0031] In a further advantageous embodiment of the method, which differs from the preceding embodiments with respect to the identification of the evaluation area in the imaging unit's image, it can advantageously be provided that an evaluation of the optical imaging properties of the imaging unit and the extrinsic calibration data relating to the imaging unit are used to determine at least one evaluation area. This has the advantage that, compared to the identification of the evaluation area described above based on an optical comparison with a reference image, only information relating to the imaging units, namely extrinsic calibration data of the imaging unit and intrinsic imaging data, i.e., optical imaging properties of the imaging unit, need to be known.In other words, this means that the knowledge of the exact positioning and orientation of the camera, together with the knowledge of the optical imaging properties of the imaging unit, is sufficient to determine in the imaging unit's image the evaluation areas, which depict parts of the laboratory work area, are located in the target storage positions.

[0032] The assignment of evaluation areas and target storage positions described here can be determined via existing intrinsic and extrinsic parameterization of the imaging unit, thereby allowing a computational positioning of the object depicted in the image, in particular the object to be recognized, in the two-dimensional image.

[0033] This allows the evaluation areas, or at least one evaluation area, to be identified quickly and reliably, even though, compared to the previously described embodiment, the presence of an object is not initially detected in the present embodiment. However, this can be achieved particularly advantageously by other means, which will be described, for example, in the following embodiments.

[0034] According to a further particularly preferred embodiment of the method, it may be provided that the automatic object recognition with the classification unit comprises a staggered or nested multiple application of the classification methods already described.It can therefore be provided that the automatic object recognition includes at least one further classification procedure based on feature generation and subsequent assignment to an object class with the features generated by the feature generation of the classification pre-stage with a set of operators trained for object recognition, wherein, based on an object class identified in the respective preceding classification procedure with a selection unit, a sub-evaluation area in the evaluation area is identified using information about the object class stored in a memory device, and the classification procedure is preferably carried out by the classification unit of the system with a set of operators specifically trained for the object class, so that a subclass of the object class becomes recognizable.In other words, this means that, on the one hand, it simplifies the respective operators and, on the other hand, enables faster and more accurate object recognition if, in a method for automatic object recognition according to the present invention, an object is first recognized according to a first coarse object class and subsequently, by means of subsequent classification procedures carried out by the classification unit, appropriately cascaded or nested subclasses of the object are recognized.

[0035] These subclasses can be defined or developed with appropriately trained operators and thus be so detailed that, in addition to the object itself, its state can also be recognized. For example, for individual laboratory items consisting of individual components, nested and / or cascaded classification procedures can determine or recognize which components or subcomponents the item is composed of and in what configuration the composition exists. It is particularly advantageous in this context if the subsequent or subordinate classification procedure does not have to process or classify the initially defined or read evaluation area, but rather if, at least in relation to the original evaluation area, it can process sub-evaluation areas, possibly with limitations.This also allows for the use of cascaded or nested sub-evaluation areas. This significantly improves recognition speed, even though it enables the recognition of objects not only in their basic form, but also in great detail, down to the individual states of individual object elements. The recognition of a tip holder or a tip holder can serve as an example. For instance, a first multi-stage classification process can identify the object class "tip holder" or "tip holder." Subsequently, a classification process can identify the specific manufacturer or type of tip holder or tip holder.By selecting one or more appropriate specialized sub-evaluation areas, the classification unit can, for example, determine in a subsequent multi-stage classification process whether and at which positions on the tip holder pipetting tips are located. Finally, the classification unit could potentially use one or more further multi-stage classification processes to determine which of the existing pipetting tips contain a liquid, which liquid, and at what fill level. Similarly, a first classification process could detect the basic presence of an object in an evaluation area.

[0036] According to a further particularly advantageous embodiment of the method, it can be provided that, following automatic recognition of an object, a class of objects or a subclass of objects, three-dimensional spatial information about the object, the class of objects or the subclass of objects stored in the system is identified, and that, using a back-processing unit, a projection of patterns extracted from the three-dimensional information of the recognized objects, class of objects or a subclass of objects is carried out into a two-dimensional imaging coordinate system, and subsequently the coordinates of the projected patterns are compared with corresponding pattern coordinates of the image, in particular the classification pre-stage, found by means of image processing.This allows information about the position and orientation of the detected objects to be obtained. This information can then be used accordingly. For example, in cascaded automatic object recognition, an incorrect orientation of an object within a particular object class can lead to the termination of further automatic recognition. The information obtained, or the comparison result, especially when deviations are detected, can also be output as a signal via an output unit and preferably transmitted to a laboratory device and taken into account during the execution and / or documentation of a laboratory routine. This embodiment also offers a particularly advantageous combination of automatic object recognition using a classification procedure performed by the classification unit on the one hand, and classical image analysis or evaluation on the other.This is because the coordinates of classically recognized patterns in the image or classification pre-stage can be compared with correspondingly calculated coordinates. These calculated coordinates are derived from a reverse calculation of known information about a recognized object and a corresponding projection into a common coordinate system. This information, along with the automatic recognition of an object, object class, or object subclass, can be used to further optimize or perfect object recognition.On the other hand, the combination of information obtained from the image and the automatically recognized objects, object classes, or object subclasses can be used, and in particular linked, to obtain not only information about the presence or existence of a specific object, but also information about its position and / or orientation. For this purpose, it is advantageous to draw on information stored in a memory or storage device for objects recognizable by the multi-stage classification procedure, information relating to three-dimensional or spatial properties, in particular dimensions and / or characteristic shapes or patterns of the objects. This allows a comparison between measured pattern coordinates based on the image and projected data from the back-calculation of the properties of the recognized, known objects.The comparison allows conclusions to be drawn, for example, about a tilted arrangement of the detected object, an uneven placement of the detected object, an incorrect or unintentional orientation of the object on the laboratory work area, especially in the area of ​​the intended placement position, and much more. This information can then be taken into account by the laboratory equipment, particularly the system's laboratory equipment, during the execution of a laboratory routine to ensure its safe and reliable execution. For example, if a laboratory object is tilted or unevenly positioned in the area of ​​its intended placement position, the laboratory equipment can issue a warning before and / or during the execution of the laboratory routine.On the other hand, the laboratory equipment could also take into account the uneven or tilted placement of the identified laboratory item when processing the laboratory routine, thus ensuring a safe and reliable execution of the laboratory routine.

[0037] According to a further particularly preferred embodiment of the method, it can be provided that the imaging unit images an area of ​​a laboratory work area which comprises an arrangement, preferably matrix-shaped, in particular a 3x3 arrangement, of target storage positions.

[0038] A further particularly preferred embodiment of the method provides that the image includes color information, in particular color information in individual color channels and preferably brightness information in a brightness channel, wherein, within the framework of automatic object recognition, the color information of the individual channels is subjected to feature generation, wherein, in particular, the channels are examined individually and the results of the examination are combined for feature generation. This allows the information about the object to be recognized, which is present in the image, especially in the evaluation area, to be used optimally for its automatic recognition in a particularly advantageous manner.

[0039] According to a further particularly advantageous embodiment of the method according to the invention, the output signal can be used to identify, monitor, and / or log a known laboratory routine of the laboratory equipment. This is because the automatic object recognition achieved through the method allows the information about the recognized objects to be used particularly advantageously in all phases of a laboratory routine—that is, continuously and after the routine—to carry out the laboratory routine safely and efficiently. For example, the mere placement of objects in the area of ​​the designated storage position of the laboratory work area can be used to identify which laboratory routines are likely to be executed.Furthermore, during the execution of a laboratory routine, it could be monitored, either at specific points or continuously, whether the detected objects, and especially their detected states, still enable or facilitate the safe execution of the laboratory routine. Finally, the automatically detected objects can ensure improved, and in particular automated, logging of completed or planned laboratory routines, for example, by automatically recording the laboratory equipment used and, if necessary, documenting its use in detail over time.

[0040] Another preferred embodiment of the method provides that the output signal is used to identify similarities to a known laboratory routine, including any deviations from the known laboratory routine of the laboratory equipment, and in particular to generate a user output relating to the similarity, preferably also the deviation. This achieves several advantages. By identifying similarities and deviations, i.e., by determining similarities and deviations, the system can, on the one hand, take a set of measures to ensure that the laboratory routine can be carried out safely or continued safely despite any deviations from a known laboratory routine. This includes, for example, changing position commands for a robot arm or otherwise adjusting control commands for parts of the laboratory equipment.In addition, a user interaction can be initiated that highlights both the similarity and the differences to a known laboratory routine, particularly based on the laboratory items positioned in the target storage positions. This user interaction can then either manually change the arrangement of the detected items in the target storage positions of the laboratory work area or require the user to confirm the execution or continuation of a modified laboratory routine adapted to the deviation.

[0041] Another embodiment of the procedure involves making the operators available via a network structure, either once or periodically, or transmitting them to the system upon a user request. Preferably, the operators are stored in an operator memory. This allows the system to be kept up-to-date with regard to the automatic recognition of objects. Furthermore, the system can be supplied with the required operators as needed and in a user-oriented manner. In addition, the automatic recognition can be improved by retransmitting revised or improved operators.

[0042] Regarding the system for the automatic recognition of laboratory work items, and in particular their condition in the area of ​​a plurality of designated storage positions of a laboratory work area, the aforementioned task is solved by a system comprising an imaging unit for generating a single two-dimensional image of a laboratory work area, an identification unit for identifying at least one first evaluation area in the two-dimensional image, wherein the at least one evaluation area is arranged and imaged in the area of ​​a designated storage position of a work area, and further comprising a processing unit for processing the image into a classification pre-stage, in particular with regard to cropping to the image of the evaluation area and / or formatting the image of the evaluation area.furthermore, a classification unit for automatic object recognition based on the classification pre-stage using at least one classification procedure based on feature generation and subsequent assignment to an object class with the features of the classification pre-stage generated by the feature generation, with a set of operators trained for object recognition, wherein the system preferably has an operator memory in which the respective operators, in particular the respective sets of specially trained operators, are stored, and also an output unit for generating an output signal identifying the automatically recognized object, as well as a communication unit for transmitting the task signal, in particular to a laboratory facility, in particular a laboratory facility included by the system.preferably for taking the identified object into account when processing and / or documenting a laboratory routine of the laboratory facility.

[0043] The system according to the invention realizes essentially the same advantages and implements the same inventive concept as the method described and disclosed above. Accordingly, reference is made to the description of the method with regard to the inventive concept and the advantages thereby realized. A key component of the system is therefore the classification unit and the operator memory for storing the specially trained operators for object recognition. This is because, on the one hand, the classification unit makes it possible to reliably and accurately identify objects, particularly laboratory equipment, without requiring a static or inflexible specification of the object to be recognized.At the same time, the system's operator memory enables reliable automatic object recognition in the first place, since the trained operators form the basis for the classification unit's function. Accordingly, the system's operator memory can be particularly advantageously designed or configured to be supplied with extended, improved, or new operators, and in particular to be written to, in order to enable continuous improvement of the system, especially its expansion to include new objects to be recognized and an improvement in object recognition.

[0044] The imaging unit can be a commercially available imaging unit from the consumer segment. The identification unit can essentially be provided by a data processing structure, for example, a data processing unit. The processing unit, the classification unit, and the output unit can also all be embedded or implemented in a data processing system, particularly in a data processing unit. The same applies to the system's communication unit, which is designed, for example, as a data interface, preferably a standardized data interface.

[0045] An advantageous embodiment may provide that the classification unit comprises a neural network unit (111) with which, within the framework of the classification procedure, at least one-stage or multi-stage classification procedure can be carried out on the basis of a convolution of the image contents of the classification pre-stage taking place at least in one stage with a set of operators trained for object recognition, in particular convolution operators.

[0046] According to an advantageous embodiment of the system, a transformation unit can be included which first transforms images generated by the imaging unit, taking into account the imaging properties of the imaging units, in particular by manipulating them perspectively. This allows the other units of the system to operate with fewer resources in terms of computing power and memory requirements. Furthermore, this can increase the recognition reliability of the objects by the classification unit.

[0047] According to an advantageous embodiment of the system, it can be provided that it has a referencing unit which can detect changes in the images over time. This referencing unit can be used to ensure the basic presence and / or the basic removal of objects in sub-areas of the images.

[0048] According to a particularly advantageous embodiment of the system, a detail recognition unit can also be included, which, following automatic object recognition by the classification unit, performs a geometric evaluation, in particular an automatic recognition of basic geometric structures, preferably the recognition of circular structures of the image, especially of the evaluation area of ​​the image, so that the detail recognition unit performs detailed recognition with respect to the recognized object. Accordingly, the detail recognition unit can be designed, for example, to recognize pipetting tips or pipette tips arranged in a recognized tip holder. Likewise, a tilt, as well as an incorrect or rotated positioning of the recognized object, can be detected. According to a further embodiment, the imaging unit can be eccentric with respect to...The system is positioned within the area of ​​the laboratory workspace captured by the imaging unit, such that the optical imaging properties of the imaging unit and its extrinsic calibration data can be used to determine at least one evaluation area. This allows the identification of at least one evaluation area without requiring a comparison with any kind of reference image. In this configuration, the at least one evaluation area is determined solely based on its position within the laboratory workspace.

[0049] In an advantageous embodiment of the system, it may also be provided that the system includes a selection unit which, based on an object class identified in a previous classification procedure, selects a sub-evaluation area within the evaluation area using information about the object class stored in a memory device, in order to perform further automatic object recognition with at least one further classification procedure based on feature generation and subsequent assignment to an object class with the features generated by the feature generation of the image content of the classification pre-stage, in particular the sub-evaluation area with a set of operators trained for object recognition.The classification procedure is preferably carried out by the classification unit (11) of the system (1) using a set of operators specifically trained for the object class, such that a subclass of the object class becomes identifiable. The selection unit thus forms the basis for nested or cascaded automatic object recognition with the classification unit of the system. Furthermore, according to the present embodiment, the system requires additional memory or memory area in which the respective partial evaluation areas for a recognized object class or object subclass are stored. In addition, the operator memory, if provided, must also be equipped with suitable, in particular suitably trained, sets of operators in order to achieve cascaded automatic recognition of an object, possibly down to details or states of the respective object.

[0050] Another particularly advantageous embodiment of the system includes a feedback unit. This unit enables the identification of three-dimensional spatial information about an object, object class, or object subclass stored in the system following automatic recognition. Furthermore, the feedback unit uses this three-dimensional spatial information to project patterns extracted from the recognized objects, object classes, or object subclasses into a two-dimensional coordinate system. Finally, the coordinates of the projected patterns are compared with corresponding pattern coordinates of the image, determined through image processing.In particular, the comparison result can be compared with the classification pre-stage. Preferably, the comparison result, especially when deviations are detected, can be output as a signal via the output unit and preferably transmitted to a laboratory facility and taken into account during the processing and / or documentation of a laboratory routine.

[0051] This allows the beneficial effects of classical image analysis methods or classical image analysis to be combined with the advantages of automatic object recognition based on a classification method or on the basis of a classification unit, particularly advantageously for increased recognition reliability and for other purposes, by means of the feedback unit.

[0052] According to a further advantageous embodiment of the method, the imaging unit can be arranged such that it images an area of ​​a laboratory workspace, comprising a, in particular, matrix-shaped arrangement of, in particular, 3x3, target storage positions. This allows for the identification of at least one evaluation area particularly quickly, reliably, and with particularly high accuracy.

[0053] According to a further preferred embodiment of the system, an identification, logging, and / or monitoring unit is included, which uses the output signal to identify, monitor, and / or log a known laboratory routine of the laboratory equipment. This ensures that the automatically detected objects and the corresponding output signal are processed accordingly, thus optimally supporting the operation of the laboratory equipment through automatic object recognition.

[0054] According to a further embodiment of the system, it may also be provided that a user interface communicates a user output which, based on the output signal, outputs or communicates an identified similarity to a known laboratory routine, in particular including the identified deviations from the known laboratory routine of the laboratory facility.

[0055] Further advantageous features and details of the invention will become apparent from the following description of the exemplary embodiments illustrated with reference to schematic drawings. These drawings show: Fig. 1 a flowchart of a method according to the invention in a first embodiment; Fig. 2 a flowchart of a method according to the invention in a second embodiment; and Fig. 3 a schematic representation of a system according to the invention.

[0056] The following will refer to Fig.1 and Fig. 2 Two embodiments of the method are described, in which feature generation and the subsequent assignment of the generated features to an object class are consistently performed using artificial neural networks. However, as described above, a variety of other methods for feature generation and assignment of the identified features to an object class can be used. These can include or incorporate artificial neural networks as needed.

[0057] The method according to the invention begins in the exemplary embodiment of the Fig. 1 The process begins with a first step S1, in which a two-dimensional image of a laboratory work area is generated using the two-dimensional imaging unit. The imaging unit is arranged and oriented such that it depicts an area of ​​the laboratory work area that represents a plurality of target storage positions for laboratory work items or laboratory objects. In a subsequent step S2, a transformation, in particular a perspective manipulation of the image, can be performed using a transformation unit, taking into account the imaging properties of the imaging unit.

[0058] In a further process step S3, the identification of at least one first evaluation area can take place in the two-dimensional representation, wherein the at least one evaluation area is arranged and represented in the area of ​​a target storage position of a work area. In the example of the Fig. 1 The identification of the evaluation area is carried out via the system's referencing unit, which checks whether different image content is detected in a sub-area, particularly in a predefined sub-area of ​​the image relative to a reference image depicting a background or the laboratory area. This content should then depict objects that could potentially be automatically recognized. If, during process step S3, image content deviating from the laboratory work area is identified in the predefined sub-area of ​​the rectified image during the comparison with the reference image, the process continues in process step S4.If no deviation is detected in the predefined sub-area compared to the corresponding sub-area of ​​the reference image or to the corresponding reference image assigned to the sub-area, the procedure continues in process step S3 with the check of the next defined sub-area of ​​the image, until all predefined sub-areas of the image have been compared with corresponding reference images and the presence of other image content besides the laboratory work area itself has been determined.

[0059] For all sub-areas where a corresponding deviation from the reference image has been detected, and which have therefore been defined as an evaluation area or as the basis for an evaluation area, procedure step S4 determines whether the image content of the sub-area that does not represent or depict the working area extends beyond the boundaries of the sub-area. If this is the case, a boundary adjustment or an adjustment of the boundaries of the sub-area is carried out in procedure step S4.1 so that the image content that differs from the background or the laboratory working area of ​​the reference image is completely encompassed by the extended evaluation area.For both the original and extended evaluation areas, the subsequent process step S5 involves preparing the data for a classification pre-stage. This involves adapting the image, particularly with regard to formatting (e.g., aspect ratio) and cropping, specifically to the evaluation area. In a subsequent process step S6, automatic object recognition is performed using the classification unit, specifically the system's neural network unit. This automatic object recognition is based on the classification pre-stage and employs at least one multi-stage classification procedure. This procedure involves at least one stage of convolution of the image content of the classification pre-stage using a set of operators trained for object recognition, particularly convolution operators.Process step S6 thus delivers as a result or output value a recognized object or laboratory item, each assigned to a specific classification stage. For example, within process step S6, an object arranged in a designated storage position within a laboratory work area can be recognized as a micro-pipitation plate of a specific type and / or from a specific manufacturer.

[0060] Following process step S6, it may also be provided that in a process step S7 a geometric evaluation, in particular an automatic recognition of basic geometric structures, preferably the recognition of circular structures of the image, in particular of the evaluation area and / or the classification pre-stage, is carried out with a detail recognition unit, with which a detail recognition of the recognized object is carried out.

[0061] Based on this automatically recognized object, the system generates an output signal in process step S8 of the inventive method by means of an output unit, which identifies the recognized object.

[0062] In the next process step S9, the output signal can be transmitted to a laboratory device via a communication unit. This output signal, and thus the detected object, can then be taken into account during the execution, configuration, or processing of a laboratory routine by the laboratory device. For example, in the final process step S10, it may be possible to adjust or modify individual process steps of a laboratory routine, such as the approach position for a robot arm of a laboratory automation system, based on the detected object and, if applicable, on the detected details of that object. Alternatively, the output signal may also be used for other purposes.

[0063] In the Fig. 2 A modified procedure of the procedure will be used. Fig. 1 described, in which part of the procedure is analogous or identical to the description of the procedure of Fig. 1 is carried out. Accordingly, procedural steps that involve a largely identical procedure, e.g., the Fig. 1 The details will not be discussed again. The generation of the image in process step S1 essentially corresponds to the procedure according to the method of Fig. 1 The same applies to manipulation within process step S2. A first difference regarding the procedure occurs in process step S3, where, according to the embodiment of the procedure as described in the Fig. 2 As depicted, no comparison of parts of the image with a reference image is performed, but rather, based on the evaluation of the optical imaging properties of the imaging unit and the extrinsic calibration data of the imaging unit, at least one evaluation area is determined or identified that maps an area of ​​a target storage position of a laboratory work area. The process step S4, which verifies the completeness of the evaluation area in the embodiment of the Fig. 1 As regards, in the embodiment of the Fig. 2 This is unnecessary because the relevant evaluation areas, or at least one evaluation area, is chosen in such a way that an extension of the evaluation area is not required.

[0064] Accordingly, process step S5 immediately follows process step S3, in that, analogous to the procedure of the Fig. 1 A processing step takes place in which a classification pre-stage is created, which is adapted, in particular with regard to the formatting of the representation of the evaluation area, for example with regard to the aspect ratio, or optimized for subsequent automatic object recognition. Following this, in process step S6, the automatic object recognition takes place with the classification unit, in particular the neural network unit of the system. In contrast to the process of Fig. 1 However, in the Fig. 2 A corresponding multi-stage classification procedure is executed multiple times in the respective process steps S6.1, S6.2, and S6.3, in particular in a cascaded manner. Upon successful recognition, an object class is identified in the multi-stage classification procedure of process step S6.1, and a plurality of object subclasses stored in the system are identified. In process step S6.2, the object subclass is again identified using the classification procedure, but now possibly with optimized or specialized operators. If further subclasses exist or are defined, an additional subclass or object subclass is automatically identified in process step S6.3. In process steps S6.1, S6.2, and S6.3...In addition to operators optimized for the already recognized object class, optimized sub-evaluation areas of the automatic object recognition can also be used as a basis for step 3. Further cascaded multi-stage classification procedures can follow process step S6.3, which can be subdivided to such an extent that, in addition to a very detailed result or automatically recognized object, results regarding the object's state can also be obtained. For example, the configuration of subcomponents of an object can be determined in a correspondingly extensive hierarchical or cascaded automatic recognition process within process step S6. It can also be provided that process steps S6.2 to S6.3 are followed by further steps.A feedback loop is then performed in process step S11, or in process steps S2 to S5, or in the execution of a similarly designed process step, which serves to determine and / or process an optimized evaluation area or sub-evaluation areas based on the recognition result of the preceding automatic object recognition S6.1 to S6.n-1. The respective results, as the optimized evaluation areas or sub-evaluation areas, and, if applicable, the optimized operators, in particular convolution operators, are made available to the subsequent automatic recognition processes S6.2 to S6.n or form input signals for the corresponding automatic recognition processes. Thus, process step S11 is not shown graphically in the diagram, although this is not done for the sake of clarity. Fig. 2 This is clarified by the successful positive or automatic detection of an object, object class, or subclass in step S6.1 to S6.n-1. The process steps S6.1 to S6.n can also include other feature generation methods and assignment methods for the detected features. Thus, the process steps S6.1 to, where applicable, S6.n also replace those described in the procedure outlined above. Fig. 1 necessary or meaningful detail recognition of process step S7. Thus, in the Fig. 2 proceeded directly with process step S8 as well as process steps S9 and S10, which in turn are largely identical or analogous to the corresponding process steps according to the procedure of the Fig. 1 expire.

[0065] A further or additional process step S6.12 to S6.1k may be provided to identify, using a feedback unit following the automatic recognition of an object, object class, or object subclass, three-dimensional spatial information about the object, object class, or object subclass stored in the system. The feedback unit then uses this known three-dimensional spatial information of the recognized object, object class, or object subclass to project patterns extracted from the three-dimensional information into a two-dimensional coordinate system. Subsequently, the coordinates of the projected patterns are compared with corresponding pattern coordinates of the image, particularly the classification pre-stage, determined by image processing. This follows process step S6.12 to S6.1k.The comparison result, particularly when deviations are detected, can be output as a signal via an output unit within process step S8. This output signal can then be considered in process step S10 when performing or processing a laboratory routine. Process steps S6.12 to S6.1k can also serve as a termination criterion for the subsequent higher-level process steps S6.2 to S6.n. This is because if a corresponding deviation is detected in process S6.12 to S6.1k, the correct definition of a specific sub-evaluation area in the mapping or the classification pre-stage may be difficult or impossible.

[0066] Fig. 3 Figure 1 shows a schematic representation of a system 1 according to the invention for the automatic detection of laboratory work items, and in particular their condition, in the area of ​​a plurality of designated storage positions 2 of a laboratory work area 3. The laboratory work area 3 and the designated storage positions 2 located therein can, for example, be part of a laboratory setup 4, such as a laboratory automation system. Furthermore, the system 1 comprises an imaging unit 5, which can, for example, be designed as a digital camera and which can be arranged either statically with respect to the laboratory work area and / or the laboratory setup 4, or can be movably arranged with respect to the laboratory work area 3 or the laboratory setup 4.In a movable implementation of the imaging unit 5, it can be particularly advantageous to provide that it moves, or can be moved, together with an object of the system, which allows for precise position determination of the imaging unit despite its fundamental mobility. For example, the imaging unit 5 can be arranged on a movable robot arm 6, whereby the exact positioning of the imaging unit 5 relative to the work area 3 can be determined, or can be ascertained, via the drive mechanism of the robot arm 6. As in the . Fig. 3 As shown, system 1 can also comprise two imaging units 5. However, an embodiment with only a single imaging unit 5 is also possible. The imaging unit 5 of system 1 is connected to a computing unit 7 of the system via a suitable data connection, wherein the computing unit 7 can in turn be part of the laboratory equipment 4 or be connected to the laboratory equipment 4 via a communication unit 8. The connection between the imaging unit 5 and the computing unit 7 allows the two-dimensional images, in particular colored two-dimensional images, generated by the imaging unit 5 to be transmitted to the computing unit 7 and further processed there.

[0067] In particular, the computing unit 7 can comprise the identification unit 9, the processing unit 10, the classification unit 11, and, if applicable, the transformation unit 12, the referencing unit 13, the detail recognition unit 14, the back-processing unit 15, and other data processing units required in the process. The classification unit 11 can include the neural network unit 111. Furthermore, the computing unit 7 may include various storage units, such as the operator memory 16, as well as other storage units or storage devices required for carrying out the process.

[0068] According to a particularly preferred embodiment, the imaging unit 5 can image an area of ​​a laboratory workspace 3, which comprises an arrangement 17 of target storage positions 2. The imaging unit 5 can be arranged eccentrically relative to the laboratory workspace and, in particular, to the arrangement of target storage positions 2, in order to allow the reconstruction of three-dimensional information from the two-dimensional images of the imaging unit 5. The target storage positions 2, which, like the laboratory workspace 3, are preferably part of the laboratory equipment 4, can preferably be optically marked, in particular, optically highlighted. These markings and highlights, which occur naturally on the laboratory workspace 3, can be used for automatic extrinsic calibration of the imaging unit 5.On the laboratory equipment 4 side, the output signal generated by the processing unit 7 of system 1 is received by an input interface 18 and further processed. This further processing can, for example, influence a control unit 19 of the laboratory equipment. Similarly, the further processing can be carried out by generating user output or initiating user interaction via a user interface 20 of the laboratory equipment. The input interface 18, the control unit 19, and / or the user interface 20 can also be located externally to the laboratory equipment 4, for example, in a separate processing unit. The generated output signal does not necessarily have to be transmitted to the laboratory equipment 4.Alternatively, other processing methods may be used, for example, as part of the documentation of a laboratory routine that is carried out externally or independently of the respective laboratory facility. Bezugszeichen

[0069] 1 System 2 Target placement position 3 Laboratory work area 4 Laboratory equipment 5 Imaging unit 6 Robot arm 7 Computing unit 8 Communication unit 9 Identification unit 10 Processing unit 11 Classification unit 12 Transformation unit 13 Referencing unit 14 Detail recognition unit 15 Reprocessing unit 16 Operator memory 17 Arrangement of target placement positions 18 Input interface 19 Control unit 20 User interface 111 Neural network unit

Claims

1. Method for operating a system (1) for automatically detecting laboratory workpieces and their condition, in particular for a workpiece carrier (1) with a plurality of workpiece holders (2) and a plurality of sensors (3) for detecting the workpieces, wherein the workpiece carrier ( the area of a plurality of target storage positions (2) of a laboratory work area (3), comprising the steps of: - generating a single two-dimensional image of a laboratory work area (3) with an imaging unit (5) for generating two-dimensional images, wherein the work area (3) comprising a matrix-like arrangement of target storage positions (2); - Identifying at least one first evaluation area in the two-dimensional image with an identification unit (9) of the system (1), wherein the at least one evaluation area is located in the area of a target storage position (2) of the work area (3) and is mapped;-processing the image with a processing unit (10) of the system (1) to a classification preliminary stage with regard to cutting to size for the image of the evaluation area and / or formatting the image of the evaluation area; - Automatic object recognition based on the classification preliminary stage using at least one classification method based on feature generation and subsequent assignment to an object class with the features generated by the feature generation using a classification unit (11) of the system (1) - Generation of an output signal identifying the detected object by means of an output unit, - transmitting the output signal to a laboratory device (4) of the system (1) with a communication unit (8) of the system (1), and taking the detected object into account in the Processing of a laboratory routine of the laboratory facility (4), namely the control of a robot arm; wherein the feature generation is based on manually generated and / or machine-learned selection criteria by artificial artificial neural networks and / or the assignment of the generated features to an object class is performed on the basis of artificial neural networks; wherein the at least one classification method comprises at least one multi-stage classification method based on at least one convolution of the image contents of the classification preliminary stage with a set of operators trained for object recognition, namely convolution operators, with a neural network unit (111) takes place in one stage; whereby, when applying the multi-stage classification procedure , an object is first recognized in the first stages of the artificial neural network according to a first rough object class, and with each subsequent stage, finer structures are recognized by the respective stage, and then, with each subsequent classification methods performed by the classification unit (11), subclasses of the object are recognized that are cascaded or nested in relation to each other; wherein the image represents color information in individual color channels, and preferably brightness information in a brightness channel, wherein, in the context of automatic object recognition, at least the color information of the individual channels is subjected to feature generation, wherein the channels are examined individually and the results of the examination are combined for feature generation; wherein the subclasses are formed in such detail by the definition of the trained operators that, in addition to the object, an object state is also recognized; wherein the respective subsequent classification process is based on a cascaded or nested sub-evaluation area of the at least one first evaluation area; wherein, in a first stage of the multi-stage classification process, an object class "tip holder" is recognized; wherein, by selecting one or more partial evaluation areas in the multi-stage classification process, the classification unit recognizes whether and at which positions of the tip holder pipette tips are arranged; and wherein, in the multi-stage classification method, the classification unit (11) in the pipette tip holder.

2. Method according to one of the preceding claims, characterized in that that, in order to determine the at least one evaluation area, an evaluation of the optical imaging properties of the imaging unit (5) and extrinsic calibration data relating to the imaging unit (5) is used.

3. Method according to one of the preceding claims, characterized that the output signal is used to identify and / or activate a known laboratory routine of the laboratory device (4). monitor and / or log a known laboratory routine of the laboratory facility4. Method according to one of the preceding claims, characterized in that that the output signal is used to determine a similarity to a known laboratory routine, in particular including the deviations from the known laboratory routine of the laboratory facility (4), and, in particular, to generate a user output relating to the similarity, preferably also the deviation.

5. System for automatic recognition of laboratory work items and their status in the area of a plurality of target storage positions (2) of a laboratory work area (3), comprising: - an imaging unit (5) for generating a single two-dimensional image of a laboratory work area (3), wherein the work area (3) comprises a matrix-like arrangement of target storage positions (2); - an identification unit (9) for identifying at least one first evaluation area in the two-dimensional image, wherein the at least one evaluation area is arranged in the area of a target placement position (2) of the work area (3) and is mapped; - a processing unit (10) for processing the image into a classification preliminary stage with regard to cropping to the image of the evaluation area and / or formatting the image of the evaluation area; - a classification unit (11) for automatic object recognition based on the classification preliminary stage using at least one classification method based on feature generation and subsequent assignment to an object class with the features generated by the feature generation with a set of operators trained for object recognition, wherein the feature generation is performed on the basis of manually generated and / or machine-learned selection criteria by artificial neural networks and / or the assignment of the generated features to an object class is performed on the basis of artificial neural networks. networks; - an output unit for generating an output signal identifying the recognized object; - a communication unit (8) for transmitting the output signal to a laboratory facility (4) of the system (1) for taking the recognized object into account when processing a laboratory routine of the laboratory device (4), namely a control signal for a robot arm; wherein the classification unit (11) comprises a neural network unit (111) with which, within the framework of the classification process at least a multi-stage classification process can be carried out on the basis of at least one stage of convolution of the image content of the classification pre-stage with a set of operators trained for object recognition, namely convolution operators; where, when the multi-stage classification method is used, an object is first recognized in the first stages of the artificial neural network according to a first rough object class, and with each subsequent stage, finer structures are recognized by the respective stage and subsequently, subclasses of the object that are cascaded or nested relative to one another are recognized by means of classification methods performed in succession by the classification unit (11); wherein the image represents color information in individual color channels, and preferably brightness information in a brightness channel, wherein, in the context of automatic object recognition, at least the color information of the individual channels is subjected to feature generation, wherein, in particular, the channels are examined individually and the results of the examination are combined for feature generation; wherein the subclasses are formed in such detail by the definition of the trained operators that, in addition to the object, an object state is also recognized; wherein the respective subsequent classification process is based on a cascaded or nested sub-evaluation area of the at least one first evaluation area; wherein, in a first stage of the multi-stage classification process, an object class "tip holder" is recognized; wherein, by selecting one or more sub-evaluation areas in the multi-stage classification method by the classification unit, it is recognized whether and at which positions of the tip holder pipette tips are arranged; and wherein, in the multi-stage classification method, the classification unit (11) in the multi-stage classification process, it is also determined in which of the pipette tips present a liquid is present, which liquid is present and at what fill level.

6. System according to one of the system claims, characterized by an identification (9) and / or logging and / or monitoring unit, which uses the output signal to identify and / or monitor and / or log a known laboratory routine of the laboratory device (4).

7. System according to one of the system claims, characterized by a user interface (20) with which, based on the output signal, an identified similarity to a known laboratory routine, in particular including the identified deviations from the known laboratory routine of the laboratory facility (4), in the form of a user output.