Laboratory container handling

Abstract

Aspects relate to a computer-implemented method, a computer program, a computer readable medium, a data processing system, and a laboratory instrument. The computer-implemented method comprises: obtaining a first image depicting at least a portion of a laboratory container; generating vector data using a first machine learning model and the first image; and using the vector data and a similarity measure, determining, by means of a first rejection algorithm, whether the laboratory container depicted in the first image belongs to one of a plurality of laboratory container classes or whether the laboratory container depicted in the first image does not belong to any of the plurality of laboratory container classes.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to, and is the non-provisional application of, provisional patent application EP23210635.1 entitled "Laboratory Container Processing", filed on 17 November 2023 with the European Patent Office. The entire application is incorporated herein by reference.

[0003] manual

[0004] The technical field is laboratory operations. More specifically, the aspects involve handling containers in a laboratory setting.

[0005] When conducting clinical tests on biological samples using laboratory instruments, it may be desirable to automate the processing of biological samples within the laboratory as much as possible.

[0006] Typically, it may be necessary to capture images of the entire container from multiple angles to determine its container category, such as classifying a container (e.g., a tube containing a biological sample) into a container category. Specifically, these images can be captured while the container is rotated. Furthermore, it may be necessary to determine several characteristics of the container, such as its height, length, width, diameter, closure shape (e.g., one closure may taper from bottom to top, while another may have a similar width from bottom to top), indentations or protrusions on the closure surface (e.g., one closure may have protrusions / indentations only on one side, while another may have protrusions / indentations on both sides or on the top), and closure color, in order to classify the container into a container category. Depending on the viewing angle, some characteristics of the closure may be more easily determined. For example, some characteristics of the closure may be more easily determined from a side view of the closure, while other characteristics may be more easily determined from a top view of the closure.

[0007] Specifically, a closure is a lid applied to a container. Determining whether a container can be classified into a container category may involve referring to a set of verification examples, where each verification example in the set is a typical representation of the corresponding container category. Each verification example may correspond to an image representing a container of the corresponding container category. If any of the determined characteristics does not match a corresponding characteristic of a verification example in the verification examples, it can be determined that the container cannot be classified. Therefore, conventional methods may lack flexibility in determining whether a container can be classified. In some cases, it may be necessary to process the container's label and / or machine-readable code in order to classify the container.

[0008] Therefore, it may be desirable to distinguish between containers that can be classified and those that cannot, particularly by doing so in a more flexible and / or more accurate manner than conventional methods. Furthermore, it may be desirable to efficiently and accurately classify containers into container categories. Additionally, it may be desirable to perform these tasks using as few images as possible, for example, using a first image that includes limited information about the container; for example, the first image may not depict the container from multiple angles or may not depict the entire container. It may also be desirable to classify containers without referring to container labels or machine-readable code.

[0009] Therefore, the purpose of this disclosure is to provide a method for processing images in a manner that allows determination of whether a container belongs to one of a plurality of container categories or not to any of these container categories.

[0010] According to a first aspect, a computer-implemented method is provided. The method includes: obtaining a first image depicting at least a portion of a container; generating vector data using a first machine learning model and the first image; and using the vector data and a similarity measure, by means of a first rejection algorithm, determining whether the laboratory container depicted in the first image belongs to one of a plurality of laboratory container categories or whether the laboratory container depicted in the first image does not belong to any of the plurality of laboratory container categories.

[0011] Specifically, the method can use a first machine learning model, a first image, and a first rejection algorithm to determine whether the container depicted in the first image belongs to one of a plurality of container categories or whether the container in the first image does not belong to any of the plurality of container categories.

[0012] In this disclosure, the terms "container" and "laboratory container" are used interchangeably.

[0013] The first rejection algorithm can use the output of the hidden layer and / or the output layer of the first machine learning model.

[0014] Generating vector data using a first machine learning model and a first image may include processing the input data using the first machine learning model. Specifically, the input data may include the first image, data composed of the first image, and / or data based on the first image.

[0015] Specifically, vector data can be associated with the coordinates of points in a vector space, the coordinates of which represent a first image, wherein the first rejection algorithm uses points in the vector space representing the first image. Specifically, vector data can include or be composed of numerical vectors. For example, according to this disclosure, a numerical vector can be a list of numerical values, such as a multidimensional list. Exemplarily, according to this disclosure, a numerical vector can be a matrix and / or array of numerical values. For example, vector data can include a list of data items, particularly a multidimensional list, for example, composed of such items. Specifically, vector data can include arrays and / or lists of data items, for example, composed of such items.

[0016] Vector data may include vector embeddings of a first image, for example, and may be the output of a first machine learning model. In this case, specifically, the vector embeddings may be associated with the coordinates of points in a vector space.

[0017] For example, the vector data may be the output of a hidden layer of a first machine learning model, particularly where the first machine learning model includes a neural network, particularly a convolutional neural network.

[0018] The output of the output layer of the first machine learning model may include information specifying the probability that a laboratory container in the first image belongs to each of the multiple laboratory container categories, particularly when the vector data is the output of the hidden layer of the first machine learning model.

[0019] The first machine learning model can be trained to classify containers into multiple container categories using training images of containers. Each training image of a container corresponds to exactly one of the multiple container categories.

[0020] In some examples, the first machine learning model may be a machine learning model that provides vector embeddings of images as input to the machine learning model, such as vector data composed of these embeddings, as output. In the case where the machine learning model is a neural network, this output is specifically the output of the output layer of the neural network.

[0021] Specifically, in these examples, the first machine learning model includes convolutional neural networks, particularly ResNet-18 or models based thereon.

[0022] In these examples, the first machine learning model can be trained based on negative image pairs comprising two images of different categories and positive image pairs comprising two images of the same category, particularly using Siamese neural networks. Alternatively, the first machine learning model can be trained based on triples comprising an anchor image, a negative image in a different category from the anchor image, and a positive image in the same category as the anchor image. For example, the use of triples is described in the context of FaceNet face recognition.

[0023] In these examples of training based on positive and negative image pairs, the first machine learning model can be trained on multiple image tuples. Training may include generating multiple tuples from multiple images. Specifically, during training, the first machine learning model may process each image in each image tuple within the tuple to obtain a corresponding vector embedding associated with the coordinates of a corresponding point in the vector space, such that the tuple is associated with multiple corresponding points. The coordinates of these points can be used to determine the distances between these points based on a similarity metric. Specifically, these distances can be used to train the first machine learning model such that points in the vector space associated with images of sample containers belonging to the same container category are clustered together and / or points in the vector space associated with images of sample containers belonging to different container categories are spaced apart in the vector space. In this way, specifically, the distances calculated using a similarity metric can allow it to be determined whether a laboratory container depicted in an image belongs to one of multiple laboratory container categories or not to any particular category. For example, when training with a Siamese neural network, shared weights are used to process the images of the tuples.

[0024] For example, machine learning models can be trained to increase the mismatch distance and / or decrease the match distance. Images depicting containers belonging to the same container category can be referred to as matching images or positive images. Images depicting containers not belonging to the same container category can be referred to as mismatch images or negative images.

[0025] Machine learning models can be trained, for example, by increasing the mismatch distance based on a similarity metric and decreasing the matching distance based on a similarity metric during training. Specifically, the mismatch distance is the distance between a first point and a second point in a vector space, where the images associated with the first point and the second point are non-matching images. Conversely, the matching distance is the distance between a first point and a second point in a vector space, where the images associated with the first point and the second point are matching images.

[0026] When training with triples, the mismatch distance can be the distance between the anchor image and the non-matching image (i.e., the negative image). The match distance can be the distance between the anchor image and the matching image (i.e., the positive image). Training with triples allows for the acquisition of vector data, even for features that are difficult to extract (e.g., cover textures and surfaces).

[0027] According to this disclosure, one or more preprocessing steps can be performed before training a first machine learning model. Specifically, images used as training data for training the first machine learning model can be preprocessed before being input into the machine learning model during training. One or more preprocessing steps may include masking highly variable portions of the image. A mask can be used to cover portions of the laboratory container that are unhelpful and may negatively impact the first machine learning model's processing of the image (particularly the accuracy of the first machine learning model's output). For example, misplaced labels and reflections can be covered by a mask. For example, a mask can cover all parts of the container except for its edges and dynamically stop where the lid begins. For example, the preprocessing steps may include covering as many areas of the image as possible that negatively affect the accuracy of the first machine learning model's output, such as areas with misplaced labels and reflections.

[0028] The method can be used to distinguish between containers that can be classified and those that cannot be classified. The method enables the determination of the container category of the containers depicted in the first image. Classification can be the output of a machine learning model and / or the determination using the output of the machine learning model (particularly, the determination using the output of the machine learning model directly or after further processing). Further processing of the output of the machine learning model can be at least partially the processing of the machine learning model's output by a rejection algorithm. Specifically, the determination using the output of the machine learning model can be included in the rejection algorithm.

[0029] The input to the first rejection algorithm may be based on, include, or consist of the output of the hidden layer of the first machine learning model and / or the output of the first machine learning model, particularly vector embeddings, or constitute thereof. Alternatively or additionally, the input to the first rejection algorithm may be based on, include, or consist of one or more distances, wherein each of the one or more distances is a distance based on a similarity metric and is the distance between a point in the vector space associated with a vector embedding and a corresponding class point in the set of class points in the vector space, wherein each class point in the set of class points corresponds to exactly one of a plurality of laboratory container classes.

[0030] Containers can be used for biological samples. Therefore, a container can be a receptacle capable of holding a biological sample, such as a test tube. Biological samples can be materials or specimens and can include blood, urine, tissue, cells, saliva, etc. Alternatively or concurrently, biological samples can include bodily fluids from a patient. A first rejection algorithm can include a set of conditions. If at least one of the conditions is met, the method can include determining that the container does not belong to any of a plurality of container categories, as will be explained in detail below.

[0031] Multiple container categories can correspond to the biological samples contained within the respective containers. For example, container categories can include serum, blood, plasma, and urine. Additionally, other container categories can be defined based on the substances added to the biological sample. For example, containers containing serum can be classified based on the type of clot activator included in the serum. Alternatively, plasma can be classified based on one or more anticoagulants added to it. Alternatively, multiple container categories can correspond to the characteristics of the container and / or closure. For example, the size or color of the closure can correspond to different categories, possibly combined with different container shapes or sizes. The characteristics of the container and / or closure can also correspond to the biological sample contained within the respective container. For example, a brown closure extending beyond the edge of the container combined with a 15 ml container volume can correspond to serum containing a specific clot activator.

[0032] Specifically, each of the multiple container categories indicates relevant information about the containers belonging to that category. This information may, for example, specify one or more of the following: the shape, geometry, weight, material, and permissible fill level of the containers belonging to that category; and / or information about how the containers belonging to that category should be disposed of (e.g., removing the cap, holding them with a clamp, recapping, etc.); and / or information about how to calculate the weight and volume of the containers belonging to that category; and / or information specifying the analyzer that can process (i.e., is capable of processing) the containers belonging to that category. For example, a urine analyzer may be able to process containers containing urine but not containers containing blood or plasma.

[0033] For example, for each of the multiple container categories, the corresponding information indicates how containers belonging to that container category should be handled, for example by an analyzer or ALS, and / or which analyzers can handle those containers.

[0034] Specifically, for each of the multiple container categories, the corresponding information can be stored in a database, which may be included, for example, in the memory of the computing device implementing the methods of this disclosure or in different memories.

[0035] For example, if it is determined that the container depicted in the first image belongs to one of a plurality of container categories, the method further includes (e.g., by accessing a database) obtaining corresponding information indicated by that category, and processing the container according to the corresponding information. For example, processing the container according to the corresponding information includes instructing one or more analyzers and / or ALSs to process the container according to the corresponding information. For example, processing the container according to the corresponding information includes processing the container by one or more analyzers and / or ALSs according to the corresponding information.

[0036] The first machine learning model can be implemented using a computer program that determines a mapping between an image (e.g., a first image) depicting at least a portion of a container and one of a plurality of container categories.

[0037] The first machine learning model can include neural networks.

[0038] A neural network may include an input layer for receiving inputs to the neural network, an output layer for generating outputs of the neural network, and one or more hidden layers optionally arranged between the input layer and the output layer. For example, a neural network may include a first set of hidden layers arranged between the input layer and a first hidden layer, and a second set of hidden layers arranged between the first hidden layer and the output layer.

[0039] As can be understood from the above, the hidden layer of the first machine learning model (also referred to as the "first hidden layer" in this paper) can be part of a neural network. Specifically, the hidden layer of the first machine learning model can be an intermediate or inner layer of the neural network. The output of the hidden layer can be obtained by processing the first image. Specifically, the first hidden layer can generate its output by processing the outputs of the hidden layers in a first set of hidden layers immediately preceding the first hidden layer as input.

[0040] The neural network of the first machine learning model can be trained to classify containers into multiple container categories using training images of the containers.

[0041] The first machine learning model can be based on a machine learning model that has been pre-trained to classify images. More specifically, the first machine learning model can be based on a pre-trained neural network. For example, the pre-trained neural network can be implemented based on the method disclosed by Tan et al., in "EfficientNetV2: Smaller Models and Faster Training," mid-2021. The first machine learning model can be based on one or more of the following neural networks: EfficientNetV2, YoloV8n-cls, ResNet50, MobileV3 Small, and Large.

[0042] Obtaining the first image may include capturing the first image using an image capture device or receiving, for example, an image previously captured by the image capture device. Specifically, the image capture device may be a camera and / or a video camera.

[0043] The training images for the container may include images obtained using an image capture device or another image capture device with corresponding capabilities and / or characteristics. The training images for the container may also include images obtained via data augmentation (e.g., data augmentation can be applied to images captured using an image capture device or another image capture device). For example, data augmentation may include adding noise, shifting the colors of an image by increasing values ​​on one or more channels, changing contrast and / or brightness, or other image modifications.

[0044] Specifically, each training image of a container can correspond to exactly one of multiple container categories (e.g., it can be associated with it). Therefore, for each training image, the corresponding training image can be classified into only one of the multiple container categories.

[0045] Image capture devices can be part of laboratory instruments (e.g., laboratory analyzers). Laboratory instruments can be used in clinical laboratories for laboratory testing of biological samples. Laboratory instruments can be designed to measure clinical parameters such as chemicals, cellular components, or biomarkers. Laboratory instruments can be designed to assist in one or more aspects of the diagnosis, monitoring, and treatment of a medical condition. Laboratory instruments can be included in automated laboratory systems (ALS). Laboratory instruments can be capable of communicating with other laboratory instruments and / or with other devices (e.g., computers). Laboratory instruments can be referred to as laboratory devices or laboratory equipment and can include laboratory automation. Laboratory instruments can be medical or clinical analyzers. Laboratory instruments may also include one or more pre-analytical, analytical, and post-analytical components.

[0046] Specifically, a laboratory analyzer is a laboratory instrument configured to perform one or more analytical steps (e.g., measuring one or more properties of a biological sample, such as the concentration of an analyte). Laboratory analyzers may include immunoassay analyzers, chemical analyzers, as well as identification and antibiotic susceptibility analyzers, bacteriological analyzers, molecular analyzers, blood analyzers, or urine analyzers.

[0047] Automated components can be used to move containers from one laboratory instrument to another, or to move containers from one component of a laboratory instrument to another component of the same laboratory instrument, or to another component of another laboratory instrument. Automated components may include tracks, belts, and / or container carriers configured to move biological samples.

[0048] Typically, an ALS (Automatic Level Analyzer) is an assembly comprising multiple components and a computing device, wherein the computing device is operatively connected to and configured to control each component. Components may be laboratory analyzers, pre-analytical laboratory instruments, post-analytical laboratory instruments and input / output modules, and automated components (e.g., conveyors, belts, container carriers) configured to move biological samples within a laboratory. Specifically, an ALS may include one or more subsystems, where each subsystem comprises one or more components of the ALS.

[0049] Preanalytical laboratory instruments are configured to perform one or more analytical steps on biological samples to prepare biological samples for analytical instruments(s). Preanalytical laboratory instruments may include centrifuges, decappers, recappers (e.g., for placing caps back onto containers), and / or dispensers.

[0050] Post-analytical laboratory instruments are configured to perform one or more post-analytical steps on biological samples after they have been processed by one or more laboratory analyzers. Post-analytical laboratory instruments may include one or more of a recapping device, storage device, refrigerator, and input / output device. Both pre-analytical and post-analytical laboratory instruments can be referred to as peri-analytical laboratory instruments.

[0051] An image capture device may be included in a component of an ALS. For example, the image capture device may be included in a tube identification unit (TIU) of the ALS. Specifically, the TIU is a component of the ALS that includes an image capture device, which is arranged (and / or may be arranged) at a predetermined position within the TIU. The TIU is configured to receive a tube at an identification position relative to the predetermined position, and the image capture device is configured to capture at least one image, such as a first image, of at least a portion of the tube when the tube is in the identification position and the image capture device is in the predetermined position. Specifically, the image capture device may be arranged (and / or may be arranged) at another predetermined position within the TIU. Furthermore, the image capture device is configured to capture another image, for example, when the image capture device is in another predetermined position and the tube is in the aforementioned identification position and / or the other identification position.

[0052] For example, using a first machine learning model, a first image, and at least one rejection algorithm to determine whether a container depicted in a first image belongs to one of a plurality of container categories, or whether a container depicted in a first image does not belong to any of the plurality of container categories according to the method of this disclosure, can have the advantage that the first image is generally sufficient to enable the determination. More specifically, it is generally not necessary to obtain additional images, obtain an image of the entire container, rotate the container, or use the container's label and / or machine-readable code.

[0053] Rejection algorithms can achieve rejection with relatively high accuracy, especially for classification, particularly because it does not require training a first-order machine learning model to identify containers to be rejected. For example, rejection algorithms can be used to overcome the difficulty of training a first-order machine learning model to classify containers into categories to be rejected.

[0054] The above method may be more flexible than the conventional method, especially because it does not require matching the container with the corresponding validation example.

[0055] Acquiring the first image may include generating the first image using a camera or receiving the first image from a camera. Therefore, the image capture device may be implemented by means of or using a camera.

[0056] The first machine learning model may include a convolutional neural network. Therefore, the neural network may include or be composed of a convolutional neural network. Hidden layers may be part of a convolutional neural network. The output of the hidden layer may include, be composed of, and / or be associated with vector data, which may be associated with the coordinates of points representing the first image in a vector space (also referred to herein as the "feature space"). The output of the hidden layer may, for example, be associated with the coordinates of points in a vector space representing the first image. Therefore, points in the vector space may represent the first image.

[0057] Convolutional neural networks (CNNs) can include one or more of the following neural networks: EfficientNetV2, YoloV8n-cls, ResNet50, MobileV3 Small, and Large. Furthermore, CNNs can include one or more additional hidden layers. For example, in addition to one or more of the aforementioned neural networks, a CNN can include another convolutional layer and a batch normalization layer, where the batch normalization layer normalizes the output of the convolutional layer and applies a linear transformation to obtain the output. The purpose of the other convolutional layer may be to reduce the number of dimensions in the output of the pre-trained neural network and / or improve performance. Specifically, a CNN can include a pre-trained neural network, such as the EfficientNetV2 neural network.

[0058] Another convolutional layer can have multiple filters. For example, the number of filters in the multiple filters can depend on the number of container categories in the multiple container categories. More specifically, the number of filters in the multiple filters can be approximately twice or at least twice the number of categories in the multiple container categories. In a specific case, if the number of container categories in the multiple container categories is approximately 12, then the number of filters in the multiple filters can be approximately 26 or at least 26. Similarly, if the number of container categories in the multiple container categories is approximately 20, then the number of filters in the multiple filters can be approximately 38 or at least 38. In some cases, performance can be improved by using at least 26 filters.

[0059] Convolutional neural networks can also include activation functions following batch-normalized layers (e.g., to specify whether neurons should be activated). For example, a convolutional neural network may include swish activation following batch-normalized layers. In some cases, the input to the hidden layers used by the first rejection algorithm can be provided by the activation function.

[0060] The pre-trained neural network can be further trained using additional hidden layers to classify containers into multiple container categories. For example, training can be performed in stages. In the first stage, the weights of the pre-trained neural network can be frozen to train one or more additional hidden layers. Then, the pre-trained neural network can be gradually unfrozen, unfreezing several layers at a time, possibly starting with the layer closest to the output layer and ending with the layer closest to the input layer.

[0061] The output of the hidden layer of the first machine learning model can include multiple numerical values ​​that can be interpreted as the coordinates of points in a vector space. More specifically, the output of the hidden layer can be normalized using a normalization algorithm (e.g., to scale the output of the hidden layer to the unit norm), and the output of the normalization algorithm can be interpreted as the coordinates of points in the vector space that contain features (e.g., numbers). The vector space can be a multidimensional space. For example, the vector space can be a Euclidean space, such as R. p , where p is an integer greater than one and equal to the number of values ​​included in the output of the hidden layer. For example, if the output of the hidden layer is in vector space R p The coordinates of points are represented by a standard basis. A vector space can be a hypersphere of Euclidean space, for example, with R... p The unit hypersphere is centered at the origin.

[0062] Specifically, in the sense that the output of the hidden layer is the coordinates of a point in vector space, the output of the hidden layer can be associated with the coordinates of a point in vector space. Specifically, when a machine learning model processes a first image, the coordinates of a point in vector space (i.e., the output of the hidden layer) represent the first image in the sense that the coordinates are the output generated by the hidden layer of a machine learning model, such as a (convolutional) neural network.

[0063] The first rejection algorithm can use a set of class points in a vector space, where each class point in the set corresponds to exactly one of a plurality of laboratory container classes.

[0064] Determining whether a container depicted in a first image belongs to one of a plurality of container categories or does not belong to any of the plurality of container categories may include obtaining a set of category points, specifically obtaining the corresponding category points representing the container category for each of the plurality of container categories.

[0065] The corresponding category point representing the container category can be one of a plurality of points representing that container category. Specifically, determining whether the container depicted in the first image belongs to one of the plurality of container categories or does not belong to any of the plurality of container categories includes obtaining the corresponding plurality of category points representing the container category for each of the plurality of container categories.

[0066] Training images can be normalized, for example, so that the pixel values ​​of each training image are between 0 and 1.

[0067] The first rejection algorithm can use a first distance, which is the distance between a point representing the first image in the vector space and a first category point in the set of category points, based on a similarity metric. The first category point represents the laboratory container category that is closest to the point representing the first image in the vector space.

[0068] Using the category point that is closest to the point representing the first image in the vector space may be advantageous in a rejection context, because even if the closest category point is not a match (e.g., exceeding a threshold distance from the point representing the first image in the vector space), it can be easily determined that no other match exists, or that the closest category point can be used to classify into the corresponding category in other ways.

[0069] In the example, determining whether a laboratory container depicted in a first image belongs to one of a plurality of laboratory container categories or not belongs to any of the plurality of laboratory container categories may include: obtaining a first distance; determining that the laboratory container does not belong to any of the plurality of laboratory container categories if at least one condition in a first set of conditions of a first rejection algorithm is satisfied; and determining that the laboratory container category of the laboratory container depicted in the first image is represented by the corresponding category point whose coordinates are closest to the point representing the first image according to a similarity metric. The first set of conditions of the first rejection algorithm may include a condition that the first distance is greater than a first threshold. Specifically, at least one condition includes a condition that the first distance is greater than the first threshold.

[0070] Specifically, a container may be considered to belong to one of a plurality of laboratory container categories if it does not satisfy any of the predefined conditions in the first set of conditions that can be regarded as rejection conditions. On the other hand, a container may be rejected because it does not belong to any laboratory container category if it satisfies one of the rejection conditions, in particular even if only a single rejection condition is met. Rejection conditions may include a first distance greater than a first threshold, or, for example, even the nearest category point is too far from the point representing the first image. In some cases, this rejection condition alone may be sufficient to reject the container because it does not belong to any laboratory container category.

[0071] The first rejection algorithm can use a second distance, where the second distance is the distance between a point representing the first image in the vector space and a second category point in the set of category points, the second category point representing the laboratory container category of the second nearest point representing the first image in the vector space.

[0072] Determining whether the laboratory container depicted in the first image belongs to one of a plurality of laboratory container categories, or whether the laboratory container depicted in the first image does not belong to any of the plurality of laboratory container categories, may include obtaining a first distance and obtaining a second distance. If at least one condition in the second set of conditions of the first rejection algorithm is satisfied, it can be determined that the laboratory container does not belong to any of the plurality of laboratory container categories. If none of the conditions in the second set of conditions of the first rejection algorithm are satisfied, it can be determined that the laboratory container category of the laboratory container depicted in the first image is represented by the corresponding category point whose coordinates are closest to the point representing the first image according to a similarity metric.

[0073] The second set of conditions for the first rejection algorithm may include one or more of the following conditions: the first distance is greater than the first threshold; the distance difference between the second distance and the first distance is less than the second threshold; and the quotient of the distance difference divided by the first distance is less than the third threshold.

[0074] Similar to the above, if any of the predefined conditions in the second set of conditions that can be considered rejection conditions are not met, the container can be considered to belong to one of the multiple laboratory container categories. On the other hand, if one rejection condition in the second set of rejection conditions is met, particularly even if only a single rejection condition in the second set of rejection conditions is met, the container can be rejected because it does not belong to any laboratory container category. The second set of conditions may, for example, include a first distance exceeding a first threshold, i.e., even the closest category point is too far away. Alternatively or additionally, the second set of conditions may include distance differences based on their respective distances to points representing the first image, where the closest and second closest category points are too close together, for example, the distance difference is too small in absolute numbers or too small relative to the first distance, as defined by the second and third thresholds, respectively.

[0075] According to this disclosure, the first rejection algorithm may be based on one or more category points representing multiple container categories. More specifically, the first rejection algorithm may include: determining how the category points relate to points in the vector space representing a first image, e.g., how they are arranged relative to them; determining the distances between the category points and points in the vector space representing the first image; and / or comparing the distances between the category points and points representing the first image. Distance comparisons enable the determination of whether points representing the first image are sufficiently close to a category point and sufficiently far from other category points in order to determine a container category.

[0076] In some examples, the first rejection algorithm may be based at least on a first distance d1 and a second distance d2. The first distance may be the distance between a point P representing the first image in the vector space and a first class point C1, where the first class point represents the container class of the point representing the first image in the vector space that is closest to it. The second distance may be the distance between a point representing the first image in the vector space and a second class point C2, where the second class point represents the container class of the point representing the first image in the vector space that is second closest to it.

[0077] Specifically, the first category point is the category point with the smallest distance to point P in the vector space among the multiple category points. Specifically, the second category point is the category point with the second smallest distance to point P in the vector space among the multiple category points. The first distance and / or the second distance can be Euclidean distance. For example, if the points in the vector space are P = (x1, …, x…) p ) T Given that the first category point is C1 = (xC1,1 , …, x C1,p ) T Given, and the second category point is C2 = (x C2,1 , …, x C2,p ) T Given the first and second distances, they can be represented as follows: ; (1a) (1b)

[0078] Specifically, the superscript "T" of a p-tuple of points in a vector space indicates transpose.

[0079] Determining whether a container depicted in a first image belongs to one of a plurality of container categories or does not belong to any of the plurality of container categories may include obtaining a first distance and obtaining a second distance. If at least one condition in the second set of conditions of the first rejection algorithm is satisfied, the method may further include determining that the container does not belong to any of the plurality of container categories. If none of the conditions in the second set of conditions of the first rejection algorithm are satisfied, the method may further include determining that the container category of the container depicted in the first image is represented by the corresponding category point whose coordinates are closest to those of the point representing the first image.

[0080] For example, obtaining the first distance and obtaining the second distance may include, for instance, calculating the first distance and the second distance using equations (1a) and (1b). For example, obtaining the first distance and obtaining the second distance may include, for instance, obtaining (e.g., calculating) the corresponding category point C for each category q in a plurality of categories. q =(x Cq,1 , …, x Cq,p ) T A point P = (x1, …, x) in vector space p ) T The corresponding distance d between q ,For example, .

[0081] The second set of conditions for the first rejection algorithm may include one or more of the following conditions:

[0082] Before determining that a container does not belong to any of the multiple container categories if at least one of the conditions in the set of conditions of the first rejection algorithm is satisfied, the method may further include determining whether at least one condition in the set of conditions is satisfied.

[0083] In some cases, it may be sufficient if the set of conditions includes a first distance greater than a first threshold and a second distance whose difference from the first distance is greater than a second threshold. Specifically, the condition that the quotient of the distance difference divided by the first distance is less than a third threshold can be omitted.

[0084] The first threshold can have a value between approximately 3 and approximately 5, or it can have a value of approximately 4. The second and third thresholds can each have a value of approximately 1 or approximately 2.

[0085] Training images may include multiple sets of training images, such as initial training images (e.g., about 70% of the training images), validation images (e.g., about 10% of the training images), and test images (e.g., about 20% of the training images). The initial training images, validation images, and / or test images may be obtained via an image capture device included in the TIU.

[0086] More specifically, there can be approximately 700 to approximately 5000 training images, for example, approximately 720 to approximately 4400 training images. The training images can be evenly distributed across multiple container categories.

[0087] Training can be performed over a specified number of epochs, where each epoch involves passing training images (e.g., one training image from a set of training images) through a machine learning model (e.g., a first machine learning model). For example, training can be performed using 15 epochs and batches with a batch size equal to 16. The loss function used during training can be classification cross-entropy. The learning rate (i.e., step size) can be set to 0.001. Alternatively, training can be performed using an exponentially decreasing learning rate (LR), which decreases after each training step (e.g., after each execution of gradient descent, such as after processing each batch), for example, according to the following equation: For example, LR0 = 0.001, attenuation rate (A) = 0.9 and attenuation steps (DS) = 67.

[0088] A training step is a single gradient update that processes a number of images corresponding to a batch size; an epoch may include many training steps. Each training step may involve processing a number of images corresponding to a batch size (e.g., if the batch size is 16, then each training step may include processing 16 images). For example, if the training image set has a cardinality of 4000, and training is performed using batches of size 16, then the number of training steps in each epoch is equal to 4000 / 16 = 250.

[0089] The first, second, and third thresholds can be selected using images from a first image set and optionally a second image set. Specifically, the first image set includes test images depicting containers that have been misclassified by a trained machine learning model. The second image set includes images depicting containers that do not belong to any of the multiple categories.

[0090] For example, the first threshold (T1) can be defined as follows. Each image I in the first image set is processed using a trained machine learning model. j (j=1,…, K), thus obtaining the corresponding point P of the vector. j = (x j,1 , …, x j,p ) T For each image I j (j=1,…,K), using the corresponding point P j Calculate multiple distances d j,1 , … d j,Q , where Q is the number of categories in the multiple categories, and d j,s It is P j Category point C associated with the s-th category s =(x Ck,1 , …, x Ck,p ) T The distance between, for example .

[0091] In some embodiments, the first threshold is determined by the distance {d} j,s} j=1,…, K; s=1,…, Q The minimum distance in the equation is given, that is: .

[0092] According to the present invention, type symbol express A set G of elements , Let G denote the smallest element of set G, and This represents the maximum element of set G.

[0093] Specifically, the first threshold (T1) can be defined as follows. The distance {d} is calculated as described above. j,s} j=1,…, K; s=1,…, Q Furthermore, each image I in the second image set is processed using a trained machine learning model. m (m=1,…,T), thus obtaining the corresponding point P in the vector space. m = (x m,1 , …, x m,p ) TFor each image I m (m=1,…,T), using the corresponding point P m Calculate multiple distances e m,1 , … e m,Q , where Q is the number of categories in the multiple categories, and e m,s It is P m Category point C associated with the s-th category s =(x Ck,1 , …, x Ck,p ) T The distance between, for example . In this case, the first threshold is determined by the distance {d} j,s} j=1,…, K; s=1,…, Q and {e m,s} m=1,…, T; s=1,…, Q The minimum distance in the set {d}, i.e., the minimum distance in the set {d} j,s} j=1,…, K; s=1,…, Q and {e m,s} m=1,…, T; s=1,…, Q The smallest element of the union of the two sets is given, that is: .

[0094] The second threshold (T2) can be defined as follows. For each image I in the first image set... j (j=1,…, K), calculate the distance {d} j,s} s=1,…, Q The corresponding minimum distance δ 1,j and the corresponding second smallest distance δ 2,j Furthermore, for each image I j (j=1,…, K), calculate the corresponding distance difference D j = δ 2,j - δ 1,j The second threshold T2 is defined as the difference {D} j} j=1，...，K The maximum value in, i.e. .

[0095] According to the present invention, the type symbol {h γ} γ=1,…, Γ Let H represent a set of Γ elements, where H = {h 1, …., h Γ}, min γ=1,…, Γ (h γ Let ) represent the minimum element of set H, and max γ=1,…, Γ (h γ ) represents the largest element of set H.

[0096] Specifically, the second threshold can be defined as the difference {D} j} j=1,…,K and { E m} m=1,…,T The maximum value in the set {D} j} j=1,…,K and { E m} m=1,…,T The largest element in the union of the sets, i.e.: , Where for the second image set E m = η 2,m - η 1,m Each image I in m (m=1,…,T), where η 1,m It is the distance { e m,s} s=1,…, Q The minimum distance in and η 2,m It is the second shortest distance.

[0097] The third threshold (T3) can be defined as follows. For each image I in the first image set... j (j=1,…,K), calculate the corresponding quotient, R j = D j / δ 1,j The third threshold T3 is defined as the quotient {R}. j} j=1,…,K The maximum value in, i.e.

[0098] In some embodiments, the third threshold can be obtained by using the quotient {R} j} j=1,…,K He Shang {U m} m=1,…,T Let U be defined, where for each m = 1, …,T, U m = E m / η 1,m The third threshold T3 is defined as the quotient {R}. j} j=1,…,K , and {U m} m=1,…,T The maximum value in the set {R} j} j=1,…,K and { U m} m=1,…,T The largest element of the union of the sets is: .

[0099] As an alternative to the methods described above for determining the first, second, and third thresholds, each of the thresholds can be determined based on the variance of each container category in the training images.

[0100] Determining whether a container depicted in a first image belongs to one of a plurality of container categories or does not belong to any of the plurality of container categories may include calculating the distance between the corresponding category point and the point representing the first image in the vector space for each of the plurality of container categories.

[0101] Each category point in the category point set can be the average of the corresponding point set in the vector space. Specifically, the average can be an arithmetic mean, a geometric mean, or a weighted average. Each point in the corresponding point set in the vector space can represent a corresponding training image. The corresponding training image can depict a container of the corresponding laboratory container category corresponding to the category point. Specifically, the corresponding training image is included in the training images of containers used to train a first machine learning model, such as a neural network for training the first machine learning model.

[0102] Therefore, the corresponding category point can be the centroid or centroid of the corresponding point set in the vector space.

[0103] Specifically, training images can be divided into multiple training image sets, each of which is associated with a corresponding category among multiple classes and includes training images depicting containers belonging to the corresponding category. For each category q among the multiple classes, there is a training image set S associated with said category q. q Having a base M q Images {I1,…,I} Mq}. The category point C associated with the category q. q = (x Cq,1 , …, x Cq,p ) T It can be given by the following equation: , Where (x) q,t,1 , …, x q,t,p ) T It is processed by using a first machine learning model that has been trained. q Image I t The output of the hidden layer.

[0104] A convolutional neural network can include an input layer and an output layer. Hidden layers can receive input, and the output of a hidden layer can have fewer values ​​than the input (i.e., fewer features). The output produced by a hidden layer can be one-dimensional and / or can summarize features of a first image. Hidden layers can be pooling layers, such as global max pooling layers. Using pooling layers as hidden layers has the advantage of providing a fixed-size output regardless of the size of the input image. This provides flexibility regarding image processing (e.g., cropping), allowing the first image to be drawn as an appropriate portion of the container.

[0105] Convolutional neural networks may also include, for example, dropout layers following pooling layers. Alternatively, convolutional neural networks may include, for example, fully connected layers following dropout layers. Fully connected layers may have a softmax activation function.

[0106] Determining whether the container depicted in the first image belongs to one of multiple container categories or does not belong to any of the multiple container categories can be based on the output of the output layer of the first machine learning model.

[0107] The first rejection algorithm can be based on the output of the output layer of the first machine learning model. Specifically, the first rejection algorithm can use the output of the output layer of the first machine learning model.

[0108] The output of the first machine learning model's output layer may include information specifying the probability that a container in the first image belongs to each of the multiple container categories.

[0109] Specifically, the output includes multiple probability values, each of which is associated with a corresponding container category among multiple categories and represents the likelihood that the input belongs to that container category. For example, for each of the multiple container categories, a value between 0 and 1 can be output, where this value can be interpreted as the likelihood that the input belongs to that container category.

[0110] Determining whether a container depicted in a first image belongs to one of a plurality of container categories or does not belong to any of the plurality of container categories may include performing a first determination of whether the container depicted in the first image belongs to one of a plurality of container categories or does not belong to any of the plurality of container categories. The first determination may include using similarity measures and vector data. Specifically, the first determination may include using the coordinates of points representing the first image and the coordinates of corresponding category points representing the container categories.

[0111] Determining whether the container depicted in the first image belongs to one of a plurality of container categories or does not belong to any of the plurality of container categories may further include performing a second determination of whether the container depicted in the first image belongs to one of a plurality of container categories or does not belong to any of the plurality of container categories. The second determination may include using the output layer of a convolutional neural network.

[0112] If the first determination differs from the second determination, determining whether the container depicted in the first image belongs to one of a plurality of container categories or whether the container depicted in the first image does not belong to any of the plurality of container categories may further include determining that the container depicted in the first image does not belong to any of the plurality of container categories.

[0113] Determining whether a container depicted in a first image belongs to one of a plurality of container categories or not belongs to any of the plurality of container categories can be performed using one or more of the following: a point in vector space representing the first image, a category point representing the corresponding container category, and a distance between a point in the first image and a category point. Specifically, as described above, the determination can be performed using a first threshold, a second threshold, and a third threshold.

[0114] Alternatively, determining whether the container depicted in the first image belongs to one of a plurality of container categories or whether the container depicted in the first image does not belong to any of the plurality of container categories may include performing a third determination and a fourth determination. The third determination may include determining which of the plurality of container categories the container depicted in the first image belongs to. The third determination may be performed using a first machine learning model, the output of the hidden layers of the first machine learning model, and the first image.

[0115] In some cases, the third determination can be performed using one or more of the following: points representing the first image in the vector space, category points representing the corresponding container categories, and the distance between the points representing the first image in the vector space and the corresponding category points. More specifically, the third determination can be performed by determining the category point representing the corresponding container category that is closest to the point representing the first image in the vector space.

[0116] The fourth determination may include determining which of a plurality of container categories the container depicted in the first image belongs to by using the output of the output layer of the first machine learning model. More specifically, the fourth determination may include determining which of a plurality of container categories the container depicted in the first image belongs to by using the highest probability among the corresponding probabilities included in the output of the output layer.

[0117] In the above alternative methods, the second set of conditions of the first rejection algorithm may include a third determination that is different from the fourth determination (e.g., constituted by it).

[0118] The second determination may include: using the output of the first machine learning model, determining whether the corresponding probability is lower than a probability threshold for each of the multiple container categories, and if the corresponding probability is lower than the probability threshold for each of the multiple container categories, determining that the container depicted in the first image does not belong to any of the multiple container categories.

[0119] A typical value for the probability threshold can be approximately 0.5. For example, the probability threshold... It can be defined as follows. Each image I in the first image set is processed using a trained machine learning model. j (j=1,…, K), thus obtaining multiple probability values ​​( ,1 , … , Q ), where Q is the number of container categories among multiple categories. The probability threshold can be defined as the probability value { ,s} j=1,…, K; s=1,…, Q The maximum value in: .

[0120] If it is determined that the container depicted in the first image does not belong to any of the multiple container categories, the method may further include obtaining a second image depicting at least a portion of the container. The second image may be different from the first image. The method may further include using a second machine learning model and the second image to determine whether the container depicted in the first image belongs to one of the multiple container categories or whether the container depicted in the first image does not belong to any of the multiple container categories.

[0121] The first image may be a side view of the container and / or the second image may be a top view of the container. Alternatively, the first image may be a top view of the container and / or the second image may be a side view of the container. The second image may be a new image different from the first image. Therefore, the first image and the second image may be separately captured images.

[0122] A second machine learning model can be trained to classify containers into multiple container categories using training images of the containers. The training images of the containers used to train the second machine learning model can be different from the training images of the containers used to train the first machine learning model. For example, the first machine learning model can be trained to classify containers into multiple container categories using training images of containers depicting side views of the respective containers. The second machine learning model can be trained to classify containers into multiple container categories using training images of containers depicting top views of the respective containers.

[0123] In some examples, the second rejection algorithm may use the output of a second machine learning model, where the output of the machine learning model may include vector embeddings of the second image (e.g., constituted by it).

[0124] Determining whether a container depicted in a first image belongs to one of a plurality of container categories or does not belong to any of the plurality of container categories using a second machine learning model and a second image may include using a second rejection algorithm. Therefore, the second rejection algorithm may correspond to the first rejection algorithm described above; however, one or more of the first, second, and third thresholds of the second rejection algorithm may differ from the corresponding thresholds of the first rejection algorithm.

[0125] For example, as mentioned above, using a first machine learning model and a first image is usually sufficient to determine the container category of the container depicted in the first image. Therefore, a second image is used only when necessary. The accuracy of the first machine learning model in determining the container category can be high enough that the use of a second image is relatively rare. Thus, the container category of a container can be determined more efficiently without requiring multiple images and without referring to the container's label or machine-readable code.

[0126] In some cases, after determining that the container category of the container depicted in the first image is represented by the corresponding category point whose coordinates are closest to those of the point representing the first image, the method may further include determining a subcategory of the determined container category. Determining the subcategory of the determined container category may include using a third machine learning model and a third image to determine whether the container depicted in the first image belongs to one of a plurality of container subcategories or not to any of the plurality of container subcategories. Determining the subcategory of the determined container category may also include using a third rejection algorithm.

[0127] The third image may be different from the first image. The third image may be the same as or different from the second image. The third image may be captured separately from the first and / or second images.

[0128] The third rejection algorithm can use the output of the hidden layer of a third machine learning model. The third machine learning model may include a convolutional neural network. The hidden layer of the third machine learning model may be part of a convolutional neural network. The output of the hidden layer of the third machine learning model may be associated with the coordinates of a point in another vector space, the coordinates of which represent a third image.

[0129] In some examples, the third rejection algorithm can use the output of a third machine learning model, where the output of the machine learning model can include vector embeddings of a third image (e.g., constituted by it).

[0130] Determining whether the container depicted in the first image belongs to one of a plurality of container subcategories or does not belong to any of the plurality of container subcategories may include obtaining a corresponding subcategory point for each of the plurality of container subcategories to represent the container subcategory.

[0131] Therefore, the third rejection algorithm can be based on one or more subclass points representing multiple container subclasses.

[0132] Furthermore, the third rejection algorithm can be based on a third distance and a fourth distance. The third distance can be the distance between a point representing the third image in another vector space and a first sub-category point representing a container sub-category that is closest to a point representing the third image in the other vector space. The fourth distance can be the distance between a point representing the third image in another vector space and a second sub-category point representing a container sub-category that is second closest to a point representing the third image in the other vector space. Determining whether a container depicted in the first image belongs to one of a plurality of container sub-categories or does not belong to any of the plurality of container sub-categories can include obtaining the third distance and obtaining the fourth distance. Specifically, determining whether a container depicted in the first image belongs to one of a plurality of container sub-categories or does not belong to any of the plurality of container sub-categories includes calculating the distance between the corresponding sub-category point and the point representing the third image in another vector space for each of the plurality of container sub-categories.

[0133] For example, if at least one condition in the set of conditions of the third rejection algorithm is satisfied, the method may further include determining that the container does not belong to any of the multiple container subcategories. If none of the conditions in the set of conditions of the third rejection algorithm are satisfied, the method may further include determining that the container subcategory of the container depicted in the first image is represented by the corresponding subcategory point whose coordinates are closest to those of the point representing the second image.

[0134] The condition set of the third rejection algorithm may include one or more of the following conditions:

[0135] The fifth threshold can be calculated in a manner similar to the first threshold if necessary. The sixth threshold can be calculated in a manner similar to the second threshold if necessary. The seventh threshold can be calculated in a manner similar to the third threshold if necessary.

[0136] The third image may depict a portion of the container that is the same as the portion of the container depicted in the second or first image. However, the portion may be depicted from a different perspective. Alternatively, the third image may depict a portion of the container that is different from the portion of the container depicted in the first image, and also different from the portion of the container depicted in the second image.

[0137] A container may include physical components of the container. For example, a container may include multiple physical components, including a body and a closure, such as a lid. The physical components of the container may be removable. For example, a physical component of the container may be a closure that is removable from the container.

[0138] Specifically, the first image can be a first region of interest in a fourth image, the fourth image depicting at least a portion of the container, wherein, specifically, the fourth image depicting the container can be acquired using an image capture device employing a TIU. Exemplarily, acquiring the first image may include acquiring (e.g., capturing and / or receiving) the fourth image, detecting the first region of interest in a subsequent image, and, for example, acquiring the first image from the fourth image by cropping the subsequent image.

[0139] Therefore, in some cases, determining whether a container depicted in a first image belongs to one of a plurality of container categories or not belongs to any of the plurality of container categories can be performed using a first region of interest in a fourth image depicting the container. The first region of interest may depict a portion of the container and, for example, provide a side view of said portion.

[0140] For example, the fourth image may include multiple regions of interest, including a top view of the container and a side view of the container.

[0141] For example, one or more mirrors in the fourth image can capture the container from different angles. Therefore, one or more regions of interest can be plotted in the corresponding mirrors in the fourth image. For example, a top view can be plotted in the mirrors in the fourth image, and / or a top view can be plotted directly in the fourth image.

[0142] In some examples, the second image may be a second region of interest (e.g., a top view including at least a portion of the container) of the fourth image. Specifically, obtaining the second image may include obtaining (e.g., capturing and / or receiving) the fourth image, detecting the second region of interest in the subsequent image, and obtaining the second image from the fourth image, for example, by cropping the subsequent image.

[0143] For example, the third image may be a third region of interest (e.g., a view of another side including at least a portion of the container) of the fourth image. Specifically, obtaining the third image may include obtaining (e.g., capturing and / or receiving) the fourth image, detecting the third region of interest in the subsequent image, and obtaining the third image from the fourth image, for example, by cropping the subsequent image.

[0144] The first region of interest, the second region of interest, and the third region of interest can be different from each other.

[0145] One or more images discussed above can be processed, such as cropped, before being acquired. Cropping can be performed based on the orientation and / or position of the container relative to the gripper.

[0146] As discussed above, one or more of the first, second, and third images can be captured independently of the other images.

[0147] Optionally, one or more regions of interest (ROIs) depict the closure of the container, such as a lid. Alternatively, one or more ROIs may depict the body of the container, for example, to differentiate between different container shapes and volumes. ROIs can be selected using edge detection analysis. Edge detection analysis may include:

[0148] Apply filters (e.g., Gaussian or adaptive) to smooth the image in order to remove noise;

[0149] Find the intensity gradient of the image;

[0150] Gradient magnitude thresholding or lower bound suppression are applied to remove false responses to edge detection;

[0151] Apply dual thresholds to determine potential edges;

[0152] By tracking the edge with hysteresis.

[0153] For example, the Canny algorithm (e.g., from the OpenCV library) can be used for edge detection. Other algorithms or variations are also possible, such as variations of the Canny algorithm that include curvelet transform.

[0154] According to a second aspect, a computer program is provided. The computer program includes instructions that, when executed by a computer, cause the computer to perform the method described above.

[0155] According to a third aspect, a computer-readable medium is provided for storing a computer program according to a second aspect.

[0156] According to the fourth aspect, a data processing system including means for performing the above-described method is provided.

[0157] According to a fifth aspect, a laboratory instrument is provided that includes a data processing system according to a fourth aspect. Specifically, the laboratory instrument may include, for example, a TIU as described herein. For instance, the TIU and the data processing system communicate data with each other, such that the TIU can provide the data processing system with images captured by the TIU, such as a first image, a second image, and / or a third image as described herein.

[0158] The subject matter described in this disclosure can be implemented as a method or on an apparatus, and may take the form of one or more computer programs (e.g., computer program products). Such computer programs can cause data processing equipment to perform one or more operations described in this disclosure.

[0159] The subject matter described in this disclosure can be implemented in data signals or on a machine-readable medium, wherein the medium is contained in one or more information carriers, such as CD-ROM, DVD-ROM, semiconductor memory, or hard disk. Specifically, the disclosed subject matter can be tangibly contained in a non-transitory machine (computer) readable medium.

[0160] Furthermore, the subject matter described in this disclosure can be implemented as a system including a processor and memory coupled to the processor. The memory can be encoded with one or more programs to cause the processor to execute one or more of the methods described in this application. Other subject matter described in this disclosure can be implemented using various machines.

[0161] Details of one or more embodiments are set forth in the following exemplary drawings and description. Other features will be apparent from the specification, drawings, and claims. Attached Figure Description

[0162] Figure 1 A schematic representation of a container identification unit and a computing device according to some embodiments is shown.

[0163] Figure 2 Containers with various closures are shown.

[0164] Figure 3 A flowchart of the steps of a method according to some embodiments is shown.

[0165] Figure 4 It shows that it can be targeted Figure 3The other steps performed in step 303.

[0166] Figure 5 An exemplary machine learning model according to some embodiments is shown.

[0167] Figure 6 An example of training a first machine learning model using a Siamese neural network is shown. Detailed Implementation

[0168] The following text provides a detailed description of the examples with reference to the accompanying drawings. Various modifications can be made to the examples. Specifically, one or more elements of an example can be combined and used in other examples to form new examples.

[0169] Figure 1 Components of a container identification unit, such as a tube identification unit 100, are shown. The tube identification unit may be part of an automated laboratory system (ALS). An automated laboratory system may be part of a laboratory. The container identification unit 100 includes a gripper 102 having multiple arms 105a, 105b, 105c (e.g., four arms), a camera 107, and a light 111. The camera 107 is positioned within the container identification unit 100. The gripper 102 is configured to hold a container 101 in an identification position, for example, by using the multiple arms 105a, 105b, 105c. The multiple arms 105a, 105b, 105c may include corresponding attachment points 106a, 106b, 106c with respect to the container 101. When the container 101 is held by the gripper 105, one or more arms of the gripper 105 may contact a closure 103. The gripper 105 may be movable in one or more directions, for example, to move the container between a loading / unloading position (not shown) and an identification position. Specifically, the loading / unloading position is the position where container 101 is loaded into container identification unit 100 and where container 101 is unloaded from container identification unit 100. Specifically, camera 107 is configured to capture an image of at least a portion of container 101 when container 101 is in the identification position and image capturing device is in a predetermined position.

[0170] As shown, container 101 can be implemented as a tube (e.g., a test tube). Container 101 can have a volume between about 1 ml and about 100 ml, more specifically between about 10 ml and about 20 ml, or between about 1 ml and 10 ml. Container 101 can contain a biological sample. The biological sample can be associated with a patient receiving treatment in a medical facility (e.g., a hospital). Container 101 can be used in a laboratory to perform clinical testing on the biological sample using ALS or laboratory instruments for ALS. Container 101 can be made of one or more of a variety of materials, such as ceramics (e.g., glass) or synthetic materials such as plastics.

[0171] The container includes a closure 103 applied thereto to close the open end of the container 101. The closure 103 may be implemented as a cap, a stopper, or other means of sealing or closing the container 101. The closure 103 may be part of the container 101 (e.g., included within the container 101) and / or the closure 103 may be removable from the container 101. The closure 103 may be provided in various shapes, sizes, and colors. The closure 103 may include a top portion, a bottom portion, and a body portion between the top and bottom portions. In some cases, the closure 103 may be generally cylindrical and / or the bottom portion of the closure 103 may have a diameter slightly larger than the diameter of the container 101. The closure 103 may have a minimal diameter change from the bottom portion to the top portion. Alternatively, the diameter of the closure 103 may gradually decrease from the bottom portion to the top portion. As an alternative, the closure 103 may have a reduced diameter in the main body portion (e.g., about 1 / 3 or about 1 / 2 of the diameter of the bottom portion), which may begin at about one-third of the distance from the bottom to the top of the closure 103.

[0172] The closure 103 may include multiple ridges surrounding or partially surrounding the body portion. The ridges may be longitudinal or vertical. The closure 103 may also include one or more lateral protrusions, which may be symmetrical or asymmetrical. The closure 103 may be made of synthetic or semi-synthetic materials, including one or more polymers, such as plastics. The properties of the closure 103 may provide indications regarding the biomaterials and / or clinical testing within the container 101.

[0173] Camera 107 can capture one or more images (e.g., a first image, a second image, a third image) of a portion of container 101 and / or a portion of closure 103. Camera 107 can communicate data 12 with computing device 110.

[0174] The container recognition unit 100 may include a mirror (not shown). The mirror may be arranged within the container recognition unit 100 in such a manner that a camera can capture an image depicting a portion of the container 101, a portion of the closure 103, and at least a portion of the mirror. The mirror may be positioned and / or arranged relative to the recognition position and predetermined position of the camera 107 such that the mirror is captured in the image, and the top of the closure 103 is drawn in the mirror captured in the image. In this manner, one image captured by the camera 107 may include a directly drawn side view and a top view drawn in the mirror. A first region of interest in the image may include the side view, and a second region of interest in the image may include the top view. Therefore, the first region of interest can be identified as the first image, and the second region of interest can be identified as the second image. In this manner, a single image captured by the camera 107 can serve as the basis for the first image, the second image, and possibly a third image.

[0175] The lamp 111 can illuminate the container 101, the closure 103 and the mirror 109 so that a usable image can be generated by the camera 107.

[0176] Exemplarily, the computing device 110 includes a processor 112 (e.g., CPU, GPU, etc.) and a memory 113. The processor 112 is configured to perform the method according to the invention. The memory 113 may include main memory and secondary memory (not shown). Specifically, the secondary memory stores a computer program comprising instructions that, when executed by the processor 113, cause the computing device 110 to perform the method according to the invention. The computing device 110 may also include an input / output (I / O) interface 111 for communicating with an input / output unit, such as a screen, keyboard, touchscreen, printer, etc.

[0177] Furthermore, computing device 110 may include a network interface controller (NIC) 114 configured to connect the device to one or more networks (e.g., an intranet, the Internet, a cellular network, etc.). In other examples, computing device 110 may include multiple NICs. Specifically, computing device 110 communicates 12 with container identification unit 100 via NIC 114, for example, with camera 107 and / or a computer (not shown) included in container identification unit 100. Specifically, computing device 110 receives images (e.g., a first image, a second image, a third image) captured by camera 107. In some embodiments, computing device 110 may be part of the container identification unit, and data transmission may occur via a bus (not shown).

[0178] Figure 2Examples of container 101 together with closures 201 to 209 are shown. Closure 201 is lighter in color than closure 103 and has a diameter that prevents it from extending beyond the edge of container 101. Closure 203 is darker in color than closure 201 and has a reduced height. Closure 205 is darker in color than closure 203 and its diameter narrows from the bottom to the top. Closure 207 has a consistent diameter from top to bottom than closure 205 and extends beyond the edge of container 101. Closure 209 does not extend beyond the edge of container 101 and has a different color than closure 207. Other closure variations are also possible, such as closures that are partially included inside container 101 when applied to it.

[0179] Figure 3 A flowchart illustrating the steps of a method according to some embodiments is shown. Specifically, Figure 3 The steps described herein can be performed by computing device 110, and in some embodiments by container identification unit 100, as described above and as follows: Figure 1 The diagram is schematically shown. Step 301 includes obtaining a first image depicting at least a portion of the (laboratory) container 101, for example, a portion of the container 101 may include closure 103 or one of closures 201 to 209. In some cases, obtaining the first image may include, for example, receiving a first image captured by camera 107 from container identification unit 100 by computing device 100. In some cases, obtaining the first image may also include, for example, capturing the first image by camera 107. Exemplarily, obtaining the first image may include, for example, receiving a fourth image captured by camera 107 from container identification unit 100 by computing device 100, and processing the fourth image to determine the first image, for example, a first region of interest. Specifically, obtaining the first image may also include, for example, capturing the fourth image by camera 107.

[0180] Step 302 includes generating vector data using a first machine learning model and a first image. For example, this may include providing a vector embedding of the first image using the first machine learning model and / or obtaining the output of the hidden layer of the first machine learning model.

[0181] Vector data can be associated with the coordinates of a point in a vector space, the coordinates of which represent the first image.

[0182] Step 303 involves using vector data and a similarity metric, with the aid of a first rejection algorithm, to determine whether the laboratory container depicted in the first image belongs to one of a plurality of laboratory container categories or whether the laboratory container depicted in the first image does not belong to any of the plurality of laboratory container categories. Specifically, the first rejection algorithm uses points representing the first image in vector space.

[0183] The first rejection algorithm can use a set of class points in a vector space, where each class point in the set corresponds to exactly one corresponding laboratory container category. Specifically, each of a plurality of container categories is associated with a corresponding class point in the set of class points. For each of the plurality of container categories, the corresponding class point can be the average of the set of corresponding points in the vector space, such as the arithmetic mean, where each point in the set of corresponding points in the vector space represents a corresponding training image that depicts a container of the corresponding container category.

[0184] In some examples, the first rejection algorithm can use the output of the hidden layer of a first machine learning model as vector data. In this example, the first machine learning model can be trained to classify containers into multiple container categories using training images of the containers.

[0185] In this example, each training image of a container corresponds to exactly one of a plurality of container categories. For example, the plurality of container categories could consist of approximately 12 container categories, approximately 20 container categories, or 10 to 25 container categories.

[0186] Figure 4 A flowchart is shown that includes exemplary steps that may be included in step 303, as described above and as follows. Figure 3 As shown in the image. Figure 4 The steps described herein can be executed by computing device 110, as described above and as follows: Figure 1 As shown schematically in the diagram.

[0187] Step 305 includes obtaining a corresponding category point for each of the plurality of container categories to represent the container category. Step 305 may include accessing memory, such as memory 113 of computing device 110, to retrieve the coordinates of the corresponding category point for each of the plurality of container categories.

[0188] Step 310 includes obtaining a first distance and a second distance. More specifically, step 310 may include calculating a corresponding distance between a point representing the first image in the vector space and a corresponding category point for each of a plurality of container categories, thereby obtaining a plurality of distances. Specifically, the first distance is the smallest distance among the plurality of distances, i.e., the distance between a point representing the first image in the vector space and a first category point, wherein the first category point is the closest among the plurality of category points to the point representing the first image in the vector space. The first category point is associated with a corresponding first container category. Specifically, the second distance is the second smallest distance among the plurality of distances, i.e., the distance between a point representing the first image in the vector space and a second category point, wherein the second category point is the second closest among the plurality of category points to the point representing the first image in the vector space.

[0189] For example, the first rejection algorithm may include a second set of conditions. For instance, the second set of conditions includes a first condition and a second condition, which are based on a first distance d1, a second distance d2, a first threshold T1, and a second threshold T2. Specifically, the contents of the first condition Γ1 and the second condition Γ2 are as follows: .

[0190] The first and second thresholds can be obtained as described above. Specifically, T1 can be equal to 4, and T2 can be equal to 1.

[0191] Step 320 includes determining whether at least one condition in the second set of conditions of the first rejection algorithm is satisfied.

[0192] If at least one condition in the second set of conditions of the first rejection algorithm is met, step 330 may include determining that the container does not belong to any of the multiple container categories. In some embodiments, for example, the computing device 110 notifies the user that the container does not belong to any of the multiple container categories by using I / O interface 111. In this case, the method may include instructing the ALS to move the container 101 to the error region of the ALS.

[0193] In an optional step, if any of the conditions in the second set of conditions of the first rejection algorithm are not met, step 340 may include determining that the container category of the container depicted in the first image is the first container category, that is, the container category represented by the first category point, that is, the category point whose coordinates are closest to the point representing the first image.

[0194] The second set of conditions for the first rejection algorithm can be tested one at a time, so that once a certain condition is met, no other condition is tested, and the container is determined not to belong to any of the multiple container categories.

[0195] For example, when the second set of conditions consists of the first condition Γ1 and the second condition Γ2, condition Γ1 is checked first. If Γ1 is satisfied, i.e., d1 > T1, then condition Γ2 is not checked and step 330 is executed. If condition Γ1 is not satisfied, then Γ2 is checked. If Γ2 is satisfied, then step 330 is executed. If Γ2 is not satisfied, then step 340 is executed.

[0196] In some cases, such as when the second set of conditions consists of the first condition Γ1 and the second condition Γ2, condition Γ2 is checked first. If Γ1 is satisfied, i.e., d2 – d1 < T2, then condition Γ1 is not checked and step 330 is executed. If condition Γ2 is not satisfied, then Γ1 is checked. If Γ1 is satisfied, i.e., d1 > T1, then step 330 is executed. If Γ1 is not satisfied, then step 340 is executed.

[0197] Figure 5 An exemplary first machine learning model is illustrated. Specifically, the first machine learning model is trained to process images (e.g., a first image) depicting at least a portion of container 101 and / or closure 103 to classify the container into multiple container categories. The first machine learning model may include a pre-trained neural network 401.

[0198] In addition, the first machine learning model may include multiple additional layers 403, 405, 407, and 411 of the first machine learning model.

[0199] Layer 403 can be implemented as a convolutional layer. Specifically, convolutional layer 403. For example... Figure 4 As shown, input 415 (specifically, the output of neural network 401) is provided to convolutional layer 403. This layer 403 processes input 415 to generate one or more outputs 417. This processing may include convolving input 415 with a set of filters, each filter recognizing a feature type from input 415 and ultimately from the underlying image (e.g., a first image), and then capturing that feature type in the corresponding transformed image of that filter. For example, as a simple example, input 415 is convolved with the following filters: The captured edge output 417 can be generated from input 415.

[0200] Layer 405 can be implemented as a batch normalization layer. This batch normalization layer can be used to make the first machine learning model more stable during training. In some embodiments, during training, the batch normalization layer 405 processes the input 417 (y) associated with the training image. 1, …, y V And generate an output 419 (z) associated with the training image. 1, …, z V), where for each j=1,…,V, the following relation holds: , (2) Where μ j and These are the mean and standard deviation of the j-th component of input 417, which are associated with the training images in the batch to which the aforementioned training images belong. Parameters and δ j These are the parameters of the batch normalization layer 405 that must be set during training. The parameter ε is, for example, a small predetermined constant that can be equal to 0.001. In some embodiments, during inference, the output 419 associated with the image can be generated using equation (2), where μ j and Replaced by the moving average of the average of the batches used during training and the moving average of the standard deviation, respectively.

[0201] Layer 407 can be implemented as a global pooling layer. Specifically, pooling layer 407 generates output 409 by processing input 419 (i.e., the output of layer 405). This can be done, for example, by organizing input 419 into a set of regions and then replacing the value in each corresponding region with a single value (e.g., the maximum value or the average value of the region). The result will be output 409, whose resolution will be reduced relative to the size of the regions into which it is split. For example, if input 419 has NxN dimensions and it is split into 2x2 regions, then output 409 will have a size of (N / 2)x(N / 2).

[0202] Layer 407 can be referred to as a hidden layer or the first hidden layer. In some examples, output 409 is used by the first rejection algorithm, for example, provided as input to the first rejection algorithm.

[0203] In this example, the vector data used together with the similarity measure to determine, by means of the first rejection algorithm, whether the laboratory container depicted in the first image belongs to one of a plurality of laboratory container categories or whether the laboratory container depicted in the first image does not belong to any of the plurality of laboratory container categories may include, be based on, or constitute of, output 409 (e.g., at least a portion of output 409).

[0204] Layer 411 may be a fully connected layer that processes input 419 (i.e., the output of the first hidden layer 407) to produce output 413 of the first machine learning model. Specifically, output 413 includes multiple probability values, where each probability value is associated with a corresponding container category among multiple categories and represents the likelihood that the input belongs to the corresponding container category. For example, for each of the multiple container categories, a value between 0 and 1 may be output, where this value can be interpreted as the likelihood that the input belongs to the corresponding container category. Exemplarily, if the multiple container categories consist of N (e.g., 12 or 20) container categories, output 413 includes N probability values.

[0205] In addition to the first machine learning model, a second and a third machine learning model can also be provided. The second and third machine learning models can each include a pre-trained neural network, for example, similar to pre-trained neural network 401. Furthermore, the second and third machine learning models can include multiple additional layers, for example, similar to additional layers 403, 405, 407, and 411 in the first machine learning model. More specifically, the second and third machine learning models can each include convolutional layers, batch normalization layers, global pooling layers, fully connected layers, and an output layer. The second and third machine learning models can differ from (and from each other) the first machine learning model because they are trained using different images than the first machine learning model, and therefore have different weights and biases after training.

[0206] Figure 6 An exemplary training of the first machine learning model and the use of a Siamese neural network for generating vector data (e.g., numerical vectors) are shown. The Siamese neural network shares weights when processing two different inputs (e.g., two different input images, hereinafter also referred to as a pair of images), that is, it uses the same weights.

[0207] For example, in the case where the first machine learning model provides, for example, vector embeddings of images as input to the machine learning model, and vector data composed of such vector embeddings as output, a method such as... can be applied. Figure 6 The training is shown. If the first machine learning model is a neural network, then this output is specifically the output of the output layer of the neural network. For example, the first machine learning model may include a convolutional neural network, particularly ResNet-18 or a model based thereon.

[0208] As mentioned above, when training with a Siamese neural network, the first machine learning model can be trained based on multiple pairs of images. Specifically, negative image pairs consist of two images from different categories, and positive image pairs consist of two images from the same category.

[0209] Specifically, during training, the first machine learning model can process each image in each of the multiple pairs of images to obtain a corresponding vector embedding, which is associated with the coordinates of the corresponding point in the vector space, such that the pair is associated with a pair of corresponding points. When trained with a Siamese neural network, images in one pair of images in the multiple pairs of images are processed using the same weights.

[0210] The coordinates of these points can be used to determine the distances between them based on a similarity metric. Specifically, these distances can be used to train a first machine learning model such that points in the vector space associated with images of sample containers belonging to the same container category are clustered together and / or points in the vector space associated with images of sample containers belonging to different container categories are spaced apart in the vector space. In this way, specifically, the distances calculated using a similarity metric can allow it to be determined whether a laboratory container depicted in an image belongs to one of several laboratory container categories or not to any single category.

[0211] For example, a first machine learning model can be trained to increase the mismatch distance and / or decrease the matching distance. The machine learning model can be trained, for example, in a manner that increases the mismatch distance based on a similarity metric and decreases the matching distance based on a similarity metric during training. Specifically, the mismatch distance is the distance between a first point and a second point in the vector space, where the image associated with the first point and the image associated with the second point are mismatched images. Specifically, the matching distance is the distance between a first point and a second point in the vector space, where the image associated with the first point and the image associated with the second point are matched images.

[0212] The output of the first machine learning model can be a numerical vector of a given size. The size of the embedding (i.e., the embedding of the numerical vector) is equal to the dimension of the output layer of the Siamese neural network. According to this disclosure, the size of the numerical vector is the dimension of the vector space. Specifically, the dimension of the vector space can include between 100 and 150, more particularly between 120 and 130. For example, the dimension of the embedding space can be equal to 128. Different types of embedding networks can be used, such as ResNet-18.

[0213] For example, Figure 6The upper part provides a schematic representation of how the Siamese neural network 510 processes the positive image pairs 501a and 501b. Specifically, each image 501a and 501b in the positive image pair is input into the first machine learning model 510 to generate corresponding numerical vectors 502a and 502b, each of which is the coordinate of a corresponding point (not shown) in the vector space. A similarity metric, in this case a Euclidean metric, is calculated to obtain the matching distance 503 between the two numerical vectors 502a and 502b (i.e., between the corresponding points in the vector space). Figure 6 The lower part provides a schematic representation of how the Siamese neural network 510 processes the negative image pairs 504a and 504b. Specifically, each image 504a and 504b in the negative image pair is input into the first machine learning model 510 to generate corresponding numerical vectors 505a and 505b, each of which is the coordinate of a corresponding point (not shown) in the vector space. A similarity metric, in this case a Euclidean metric, is calculated to obtain the mismatch distance 506 between the two numerical vectors 505a and 505b (i.e., between the corresponding points in the vector space).

[0214] Numerical vectors can be used in conjunction with similarity metrics and a reference database to determine, using a first rejection algorithm, whether a laboratory container depicted in a first image belongs to one of a plurality of laboratory container categories or does not belong to any of the plurality of laboratory container categories. The reference database comprises a plurality of reference images, each of which depicts a corresponding sample container of a corresponding sample container category.

[0215] Specifically, the first machine learning model can be used to generate a numerical vector associated with the first image. The first machine learning model can also be used to obtain a corresponding numerical vector associated with each of a plurality of reference images.

[0216] Based on these numerical vectors, a first distance is determined, which is referred to below as the minimum distance. Specifically, in some examples, determining the first distance includes determining multiple distances. Specifically, this may include, for each of a plurality of reference images, determining a corresponding distance associated with it using a similarity metric, the distance being the distance between the corresponding numerical vector associated with that image and the numerical vector associated with the first image. In this case, the first distance is the minimum distance among the multiple distances and is associated with a container category, namely the container category of the container depicted in the image associated with the minimum distance.

[0217] Alternatively, determining the first distance may include determining a set of category points, each category point corresponding to exactly one corresponding laboratory container category. Each category point is the average of the set of corresponding points in a vector space, where each point in the set of corresponding points in the vector space represents a corresponding image among a plurality of reference images, which depict a laboratory container corresponding to the corresponding laboratory container category of that category point. In this case, the first distance is the distance between a point in the vector space representing the first image and a first category point in the set of category points, the first category point being the point that is closest to the point represented by the numerical vector associated with the first image according to a similarity metric. In this case, the first distance is associated with the container category (i.e., the container category corresponding to the first category point).

[0218] For the first distance, a determination can then be made, which includes determining whether the minimum distance is greater than a threshold. If the minimum distance is greater than the threshold, it can be determined that the container depicted in the first image does not belong to any of the multiple container categories. Otherwise, it can be determined that the container depicted in the first image belongs to the container category associated with the first distance.

[0219] In this example, one-time learning can be achieved. That is, when new reference data is added, for example, to expand the reference database by adding new image data, the first machine learning model does not need to be retrained. Instead, when new data is added, such as a new image depicting a laboratory container, the first machine learning model can be used to obtain vector data representing the new image. The vector data is associated with the coordinates of a point in a vector space that represents the new image. If each category point in the category point set is the average of the corresponding point set in the vector space, then the point representing the new image can be added to the point set corresponding to the category to which the laboratory container depicted in the new image belongs.

[0220] As an alternative to training with image pairs as described above in the context of Siamese neural networks, image triples and triple loss can be used, for example, as explained above. This can be based, for example, on known methods of using triple loss in face recognition, such as in the FaceNet algorithm (e.g., described in Schroff et al., “FaceNet: A Unified Embedding for Face Recognition and Clustering,” arXiv:1503.03832v3, June 17, 2015). In this case, specifically, the first machine learning model can be the machine learning model used by the FaceNet algorithm.

[0221] Other examples

[0222] 1. A computer-implemented method, the method comprising: obtaining a first image depicting at least a portion of a container; using a first machine learning model, the first image, and a first rejection algorithm, determining whether the container depicted in the first image belongs to one of a plurality of container categories or whether the container depicted in the first image does not belong to any of the plurality of container categories; wherein the first rejection algorithm uses the output of a hidden layer of the first machine learning model; wherein the first machine learning model is trained to classify containers into the plurality of container categories using training images of containers, wherein each training image of the containers corresponds to exactly one of the plurality of container categories.

[0223] 2. The method according to Example 1, wherein the first machine learning model includes a convolutional neural network, wherein the hidden layer is part of the convolutional neural network; wherein the output of the hidden layer is associated with the coordinates of a point in a feature space, the coordinates of which represent the first image.

[0224] 3. The method according to Example 2, wherein determining whether the container depicted in the first image belongs to one of the plurality of container categories or whether the container depicted in the first image does not belong to any of the plurality of container categories includes obtaining a corresponding category point for each of the plurality of container categories to represent the container category.

[0225] 4. The method according to Example 3, wherein the first rejection algorithm is based on one or more category points representing the plurality of container categories.

[0226] 5. The method according to Example 4, wherein the first rejection algorithm is based at least on a first distance and a second distance, wherein the first distance is the distance between a point representing the first image in the feature space and a first category point, the first category point representing the container category closest to the point representing the first image in the feature space, and wherein the second distance is the distance between a point representing the first image in the feature space and a second category point representing a container category second closest to the point representing the first image in the feature space.

[0227] 6. The method according to Example 5, wherein determining whether a container depicted in the first image belongs to one of the plurality of container categories or does not belong to any of the plurality of container categories comprises: obtaining a first distance; obtaining a second distance; determining that the container does not belong to any of the plurality of container categories if at least one of the conditions of the first rejection algorithm is satisfied; and determining that the container category of the container depicted in the first image is represented by the corresponding category point whose coordinates are closest to those of the point representing the first image if none of the conditions of the first rejection algorithm are satisfied, wherein the conditions of the first rejection algorithm include one or more of the following conditions: the first distance is greater than a first threshold; the distance difference between the second distance and the first distance is less than a second threshold; and the quotient of the distance difference divided by the first distance is less than a third threshold.

[0228] 7. The method according to Examples 3 to 6, wherein for each of the plurality of container categories, the corresponding category point is the arithmetic mean of the corresponding set of points in the feature space, wherein each point in the corresponding set of points in the feature space represents a corresponding training image, the corresponding training image depicting a container of the corresponding container category.

[0229] 8. The method according to any one of the foregoing examples, wherein determining whether the container depicted in the first image belongs to one of the plurality of container categories or whether the container depicted in the first image does not belong to any of the plurality of container categories is based on the output of the output layer of the first machine learning model; wherein the first rejection algorithm may be based on the output of the output layer of the first machine learning model.

[0230] 9. The method according to Example 8, wherein the output of the output layer of the first machine learning model includes information specifying the probability that a container in the first image belongs to the container category for each of the plurality of container categories.

[0231] 10. The method according to Example 8 or 9, wherein determining whether a container depicted in the first image belongs to one of the plurality of container categories or whether a container depicted in the first image does not belong to any of the plurality of container categories comprises: performing a first determination of whether a container depicted in the first image belongs to one of the plurality of container categories or whether a container depicted in the first image does not belong to any of the plurality of container categories, wherein the first determination comprises using coordinates representing points of the first image and corresponding category points representing the container categories; performing a second determination of whether a container depicted in the first image belongs to one of the plurality of container categories or whether a container depicted in the first image does not belong to any of the plurality of container categories, wherein the second determination comprises using the output of the output layer of the convolutional neural network; and determining that a container depicted in the first image does not belong to any of the plurality of container categories if the first determination differs from the second determination.

[0232] 11. The method according to a combination of Examples 10 and 9, wherein the second determination comprises: determining, for each of the plurality of container categories, whether a corresponding probability is lower than a probability threshold by using the output of the first machine learning model, and determining that the container depicted in the first image does not belong to any of the plurality of container categories if the corresponding probability is lower than the probability threshold for each of the plurality of container categories.

[0233] 12. The method according to any one of the foregoing examples, wherein if it is determined that the container depicted in the first image does not belong to any of the plurality of container categories, the method further comprises: obtaining a second image depicting at least a portion of the container, wherein the second image is different from the first image; and using a second machine learning model and the second image to determine whether the container depicted in the first image belongs to one of the plurality of container categories or whether the container depicted in the first image does not belong to any of the plurality of container categories; wherein the first image may be a side view of the container, and / or the second image may be a top view of the container.

[0234] 13. A computer program including instructions that, when executed by a computer, cause the computer to perform the method described in any of the preceding examples.

[0235] 14. A computer-readable medium storing a computer program according to Example 13.

[0236] 15. A data processing system comprising means for performing the method according to any one of Examples 1 to 12.

Claims

1. A computer-implemented method, the method comprising: Obtain a first image depicting at least a portion of the laboratory container; Vector data is generated using the first machine learning model and the first image; as well as Using the vector data and similarity metric, and with the aid of a first rejection algorithm, it is determined whether the laboratory container depicted in the first image belongs to one of a plurality of laboratory container categories or whether the laboratory container depicted in the first image does not belong to any of the plurality of laboratory container categories.

2. The method according to claim 1, The vector data is associated with the coordinates of a point in a vector space, the coordinates of which represent the first image, and the first rejection algorithm uses the point in the vector space that represents the first image.

3. The method according to claim 1 or 2, The first rejection algorithm also uses a set of category points in a vector space, wherein each category point in the set corresponds to exactly one of the plurality of laboratory container categories.

4. The method according to claim 3, Determining whether the laboratory container depicted in the first image belongs to one of the plurality of laboratory container categories or does not belong to any of the plurality of laboratory container categories includes obtaining the set of category points.

5. The method according to claim 3 or 4, Each category point in the category point set is the average value of the corresponding point set in the vector space, where each point in the corresponding point set in the vector space represents a corresponding training image, and the corresponding training image depicts the laboratory container of the corresponding laboratory container category corresponding to each category point.

6. The method according to claim 2 and any one of claims 3 to 5, The first rejection algorithm uses a first distance. The first distance is the distance between a point representing the first image in the vector space and a first category point in the set of category points, based on the similarity metric, where the first category point represents the laboratory container category that is closest to the point representing the first image in the vector space.

7. The method according to claim 6, Determining whether the laboratory container depicted in the first image belongs to one of the plurality of laboratory container categories or does not belong to any of the plurality of laboratory container categories includes: Obtain the first distance; If at least one condition in the first set of conditions of the first rejection algorithm is met, then it is determined that the laboratory container does not belong to any of the plurality of laboratory container categories; as well as If none of the conditions in the first set of conditions of the first rejection algorithm are met, then the laboratory container category of the laboratory container depicted in the first image is determined to be the laboratory container category represented by the corresponding category point whose coordinates are closest to the point representing the first image according to the similarity metric. The first set of conditions for the first rejection algorithm includes the condition that the first distance is greater than the first threshold. In particular, the at least one condition includes the condition that the first distance is greater than the first threshold.

8. The method according to claim 6 or 7, The first rejection algorithm also uses a second distance. The second distance is the distance between a point representing the first image in the vector space and a second category point in the set of category points, where the second category point represents a laboratory container category that is closest to a point representing the first image in the vector space.

9. The method according to claim 8, Determining whether the laboratory container depicted in the first image belongs to one of the plurality of laboratory container categories or does not belong to any of the plurality of laboratory container categories includes: Obtain the first distance; Obtain the second distance; If at least one condition in the second set of conditions of the first rejection algorithm is met, then it is determined that the laboratory container does not belong to any of the plurality of laboratory container categories; as well as If none of the conditions in the second set of conditions of the first rejection algorithm are met, then the laboratory container category of the laboratory container depicted in the first image is determined to be the laboratory container category represented by the corresponding category point whose coordinates are closest to the point representing the first image according to the similarity metric. The second set of conditions for the first rejection algorithm includes one or more of the following conditions: The first distance is greater than the first threshold; The distance difference between the second distance and the first distance is less than a second threshold; and The quotient of the distance difference divided by the first distance is less than the third threshold.

10. The method according to any one of the preceding claims, The vector data includes the vector embedding of the first image and is the output of the first machine learning model.

11. The method according to any one of claims 1 to 9, The first machine learning model includes neural networks, particularly convolutional neural networks, and The vector data mentioned above is the output of the hidden layer of the neural network.

12. The method according to claim 11, The output of the output layer of the first machine learning model includes information specifying the probability that the laboratory container in the first image belongs to each of the plurality of laboratory container categories.

13. The method according to claim 12, Determining whether the laboratory container depicted in the first image belongs to one of the plurality of laboratory container categories or does not belong to any of the plurality of laboratory container categories includes: Perform a first determination: whether the laboratory container depicted in the first image belongs to one of a plurality of laboratory container categories, or whether the laboratory container depicted in the first image does not belong to any of the plurality of laboratory container categories. The first determination includes using the similarity metric and the vector data; A second determination is made regarding whether the laboratory container depicted in the first image belongs to one of a plurality of laboratory container categories, or whether the laboratory container depicted in the first image does not belong to any of the plurality of laboratory container categories. The second determination includes using the output layer of the convolutional neural network; and If the first determination differs from the second determination, then it is determined that the laboratory container depicted in the first image does not belong to any of the plurality of laboratory container categories.

14. The method according to claim 13, wherein, The second determination includes: By using the output of the first machine learning model, for each of the plurality of laboratory container categories, it is determined whether the corresponding probability is lower than a probability threshold, and if the corresponding probability is lower than the probability threshold for each of the plurality of laboratory container categories, it is determined that the laboratory container depicted in the first image does not belong to any of the plurality of laboratory container categories.

15. The method according to any one of the preceding claims, in, If it is determined that the laboratory container depicted in the first image does not belong to any of the plurality of laboratory container categories, the method further includes: Obtain a second image depicting at least a portion of the laboratory container, wherein the second image is different from the first image; and The second machine learning model and the second image are used to determine whether the laboratory container depicted in the first image belongs to one of the plurality of laboratory container categories or whether the laboratory container depicted in the first image does not belong to any of the plurality of laboratory container categories. The first image may be a side view of the laboratory container, and / or the second image may be a top view of the laboratory container.

16. A computer program comprising instructions that, when executed by a computer, cause the computer to perform the method according to any one of the preceding claims.

17. A computer-readable medium storing a computer program according to claim 16.

18. A data processing system comprising means for performing the method according to any one of claims 1 to 15.

Images