Device tag placement assessment based on machine learning

Abstract

Systems and methods for machine vision based metrology in automated label placement inspection and quality control inspection are disclosed. An exemplary system may evaluate label placement on an object by applying one or more trained machine learning (ML) models to an image of the object. The ML models are trained to identify the object and tags on the object. The trained ML patterns are used to identify morphological features, such as edges of the object and the tag, and mold morphological features may be used to determine spatial metrics. Based on the spatial metric, a tag placement indicator may be generated to indicate that the tag passes the quality control check when the tag is placed within a predetermined tolerance on the object.

Description

Priority Statement

[0001] This application claims the benefit of priority to U.S. Patent Application Serial No. 63 / 586,143, filed September 28, 2023, which is incorporated herein by reference in its entirety. Technical Field

[0002] This document generally relates to metrology systems and technologies, and more specifically, but not in a restrictive way, to machine learning (ML)-based metrology for automated product inspection, including automated label placement inspection and quality control checks. Background Technology

[0003] Metrology is the science of measurement, encompassing experimental and theoretical determinations at any level of uncertainty in any scientific and technological field. Metrology using machine vision is widely used to measure distances between objects, often as a quality control measure. An example of machine vision metrology in manufacturing and packaging is measuring plunger placement in a syringe barrel. Image analysis is used to locate the top edge of the plunger and the top edge of the syringe flange. The plunger placement can then be determined by calculating the pixel-level distance between these two edges and converting that distance to real units after calibrating the imaging system.

[0004] Conventional edge detection can utilize derivative images. Gradient-based methods can be used to locate edges, calculating the slope between adjacent pixels and detecting an edge when the slope exceeds a preset threshold. The slope can be calculated by subtracting the intensities of two pixels separated by a predefined number of pixels. Such gradient-based edge detection methods typically achieve sufficient performance under several assumptions, including, for example, that the edge should be within a predefined region in the image and that the region should extend, that the edge has good and consistent contrast with the background, and that there are no other interfering edges or elements in the predefined edge search region that could potentially confuse the edge of interest. However, in many real-world cases, not all of these assumptions are fully met, and simple gradient-based edge detection methods may be less immune to noise and interference that blurs the edge of interest. The sources and manner of interference can be highly variable and sometimes unpredictable or uncontrollable. For example, the edge of interest may be obscured by other edges on an object, print, blemishes, stains, or other structural elements. Additionally, imaging modes or conditions (such as lighting or illumination conditions, zoom settings, or viewing angles) can also affect the observability and detectability of edges or other features of interest. Therefore, traditional algorithms may produce erroneous or inconsistent edges, no edges or incorrect edge locations, and inaccurate metrological results (e.g., distances between edges). For at least the above reasons, there is an unmet need in applications for more robust machine vision-based image feature detection and metrology techniques. Summary of the Invention

[0005] This application describes a machine vision-based metrology system and method. The application demonstrates a machine vision-based metrology system and method, whose applications include, but are not limited to, automated label placement inspection and quality control checks. Artificial intelligence or machine learning (ML) techniques are used to improve the accuracy and efficiency of object recognition and image feature extraction. According to one embodiment, a label placement inspection and quality control system can evaluate the label placement on one of a plurality of objects of substantially similar size and each having a label of substantially similar size. The system may include an imaging device for acquiring an image of the object or sample, a storage device for storing at least one trained machine learning (ML) model, and programmable circuitry such as a dedicated programmable microprocessor. The stored ML model can be trained to establish a correspondence between an input image and the recognition of objects from the input image, labels associated with the objects, and features of the objects or labels. The programmable circuitry can apply at least one trained ML model to the acquired image of the object to identify morphological features of the object and the label, and determine a spatial metric between the identified morphological features. Based at least in part on the spatial metric, the programmable circuitry can generate a label placement indicator to indicate whether the label placement is within a predetermined tolerance on the object. If label placement fails quality control checks, a notification or alarm can be generated and provided to the user. Label placement indicators can be used to control the robotic system to handle objects with defective, missing, or misaligned labels.

[0006] The ML-based edge detection and metrology described herein enhances the efficiency and quality assurance of automated label placement on objects or samples. Compared to simple gradient-based edge detection methods in conventional metrology performed via machine vision, the ML-based techniques described herein are less susceptible to noise, interference, poor lighting or illumination conditions, or the pose or orientation of the object or sample being inspected. This allows for more robust, accurate, and efficient detection of edges or other morphological features in object images, and thus allows for more consistent and reliable metrological measurements. When used in automated label placement inspection, the ML-based method of this invention can be SKU / label location-independent, unlike conventional methods where edge detection is specific to stock units (SKUs) and may require individual inspection for each SKU. Furthermore, the ML-based method of this invention can be implemented using a single camera, thus requiring only personnel familiar with camera software. Therefore, the efficiency of the label placement inspection and quality assurance process can be enhanced, product yield can be increased, and overall costs can be reduced.

[0007] In various examples, this subject matter relates to a system for evaluating label placement on one of a plurality of objects, each object being substantially similar in size and having labels of substantially similar size, the system comprising: an imaging device configured to acquire an image of the object; a storage device configured to store at least one trained machine learning (ML) model trained to establish a correspondence between an input image and the recognition of an object from the input image, a label associated with the object, and features of the object or the label; and programmable circuitry configured to: apply the at least one trained ML model to the acquired image to recognize morphological features of the object and morphological features of the label, and determine a spatial metric between the recognized morphological features of the label and the recognized morphological features of the object; and generate a label placement indicator at least in part based on the determined spatial metric to indicate whether the label placement is within a predetermined tolerance on the object.

[0008] In some examples, the morphological features of the object include a first edge of the object, the morphological features of the label include a first edge of the label, and the spatial metric includes the distance between the first edge of the object and the first edge of the label. The programmable circuit is configured to generate a label placement indicator that includes one or more of the following: (i) a positive placement indicator if the determined distance between the first edge and the second edge is within a predetermined range; or (ii) a misalignment indicator if the determined distance between the first edge and the second edge is outside the predetermined range.

[0009] In some examples, the object is a drug injection device that includes two interconnectable elements and a cylindrical outer portion for displaying a label, wherein the morphological features of the object include the object's seams.

[0010] In some examples, the first edge and the second edge are each represented by a corresponding set of image pixels spatially distributed at least along the longitudinal direction of the outer portion of the cylinder, wherein programmable circuitry is configured to determine a spatial metric including the distance between the centroids of the corresponding sets of image pixels representing the first edge and the second edge, respectively.

[0011] In some examples, the at least one trained ML model includes a first trained ML model and a second trained ML model, respectively, for a first edge and a second edge, wherein programmable circuitry is configured to apply the first trained ML model to the acquired image to identify the first edge, and to apply the second trained ML model to the acquired image to identify the second edge. In some examples, the first trained ML model and the second trained ML model differ in at least one of the model architecture or model parameters. In some examples, at least one of the first trained ML model or the second trained ML model is a trained deep learning model.

[0012] In some examples, the programmable circuitry includes a training module configured to generate or update an ML model via a training dataset, which comprises multiple images of objects of substantially similar size, each with labels of substantially similar size and variations in label placement on the objects. In some examples, the training module is configured to further generate or update the ML model using user-identified features from each of the multiple images in the training dataset. In some examples, the multiple images include images taken under different imaging modes or conditions, or images of objects in different orientations, poses, or rotations.

[0013] In some examples, imaging modes or conditions include one or more of zoom settings, lighting or illumination conditions, contrast settings, sharpness settings, exposure levels, or viewing angles toward the object or label.

[0014] In various examples, the multiple images include those that have been manually or algorithmically modified to blur or partially obscure the morphological features of objects.

[0015] In various examples, the at least one trained ML model includes an object detection model trained to generate at least one bounding box from the input image identifying the location of an object or the location of a label. In various examples, the at least one trained ML model includes an image segmentation model trained to determine the pixelated contours of an object or a label from the input image. Various examples include a user interface configured to generate a label misalignment alert when a label placement indicator indicates that a label is not within a predetermined tolerance on the object. Various examples include a robot controller configured to manipulate an object via a robot according to a label placement indicator and generate control signals for manipulating the object via the robot according to the label placement indicator.

[0016] In various examples, this subject matter relates to a method for evaluating label placement on one of a plurality of objects, each object being substantially similar in size and having labels of substantially similar size, the method comprising: acquiring an image of the object; receiving from a storage device at least one trained machine learning (ML) model, the ML model being trained to establish a correspondence between an input image and the recognition of an object from the input image, a label associated with the object, and features of the object or the label; applying the at least one trained ML model to the acquired image of the object via programming circuitry to recognize morphological features of the object and morphological features of the label therefrom; determining a spatial metric between the recognized morphological features of the label and the recognized morphological features of the object; and generating a label placement indicator at least in part based on the determined spatial metric to indicate whether the label placement is within a predetermined tolerance on the object.

[0017] In some examples, the morphological features of the object include a first edge of the object, the morphological features of the label include a first edge of the label, and the spatial metric includes the distance between the first edge of the object and the first edge of the label. The programmable circuit is configured to generate a label placement indicator that includes one or more of the following: (i) a positive placement indicator if the determined distance between the first edge and the second edge is within a predetermined range; or (ii) a misalignment indicator if the determined distance between the first edge and the second edge is outside the predetermined range.

[0018] In some examples, the object is a drug injection device that includes two interconnectable elements and a cylindrical outer portion for displaying a label, wherein the morphological features of the object include the object's seams.

[0019] In some examples, the at least one trained ML model includes a first trained ML model and a second trained ML model that differs from the first ML model in at least one of the model architecture or model parameters. The method includes applying the first trained ML model to the acquired image to identify a first edge, and applying the second trained ML model to the acquired image to identify a second edge.

[0020] In some examples, the method includes generating or updating an ML model via an ML training module using a training dataset comprising multiple images of objects of substantially similar size, each with labels of substantially similar size and variations in label placement on the objects. In some examples, the method includes generating or updating the ML model using user-identified features of each of the multiple images in the training dataset.

[0021] In some examples, the multiple images include images taken under different imaging modes or conditions, or images of objects in different orientations, poses, or rotations.

[0022] In some examples, imaging modes or conditions include one or more of zoom settings, lighting or illumination conditions, contrast settings, sharpness settings, exposure levels, or viewing angles toward the object or label.

[0023] In various examples, the multiple images include those that have been manually or algorithmically modified to blur or partially obscure the morphological features of objects.

[0024] In various examples, the at least one trained ML model includes an object detection model trained to generate at least one bounding box from the input image identifying the location of an object or the location of a label. In various examples, the at least one trained ML model includes an image segmentation model trained to determine the pixelated contours of an object or a label from the input image. Various examples include generating a label misalignment alert when a label placement indicator indicates that the label is not within a predetermined tolerance on the object. Various examples include generating control signals for manipulating an object by a robot based on a label placement indicator.

[0025] Although the ML-based inspection and measurement techniques described herein are discussed in the context of label placement inspection and quality control checks, such techniques can be used in many other applications for machine vision-based product manufacturing and quality assurance. The above overview is a general overview of some of the teachings of this application and is not intended to be exclusive or exhaustive of the subject matter. Further details regarding the subject matter are found in the detailed description and the appended claims. Other aspects of this disclosure will be apparent to those skilled in the art upon reading and understanding the following detailed description and viewing the accompanying drawings, each of which should not be considered limiting. The scope of this disclosure is defined by the appended claims and their legal equivalents. Attached Figure Description

[0026] Various embodiments are illustrated by way of example in the accompanying drawings. Such embodiments are illustrative and are not intended to be exhaustive or exclusive embodiments of the subject matter.

[0027] Figure 1 The label placement inspection and quality control check system for evaluating label placement on objects or samples is shown in an illustrative rather than restrictive manner.

[0028] Figure 2 Images of example features of a drug injection device and the placement of a label on a positioning device are shown in an illustrative rather than restrictive manner.

[0029] Figure 3 Images of a selected set of objects and label features for drug injection devices and label placement on positioning devices are shown as examples rather than limitations.

[0030] Figure 4 Examples of features of interest selected by the user for identification by the label placement inspection and quality control check system are presented in an illustrative rather than restrictive manner.

[0031] Figures 5A to 5B It is a diagram that shows an example of training an ML model and using the trained ML model to identify objects or labels from an image and detect features of interest, and optionally make a pass / fail quality control decision.

[0032] Figure 6 This is a flowchart demonstrating an example method for evaluating the placement of labels on an object.

[0033] Figure 7 The block diagram of an example machine is shown in general, on which any or more of the techniques (e.g., methods) discussed in this article can be executed. Detailed Implementation

[0034] This paper discloses a machine vision-based system and method for automated inspection and quality control of label placement on objects or samples. An exemplary system can use one or more trained machine learning (ML) models to evaluate label placement on objects. The ML models are trained to establish a correspondence between an input image and the recognition of objects from that input image, labels associated with those objects, and features of the objects or labels. Using the trained ML models, morphological features such as the edges of the objects and / or the edges of the labels can be detected from the images of the objects, and these morphological features can be used to determine spatial metrics. Based on the spatial metrics, label placement indicators can be generated to indicate that a label has passed a quality control check when it is placed on the object within a predetermined tolerance. If the label placement fails the quality control check, a notification or alarm can be generated and provided to the user. The label placement indicators can be used to trigger a robotic system to manipulate objects with defective, missing, or misaligned labels.

[0035] The following detailed description of this subject matter is taken with reference to the accompanying drawings, which illustrate, by way of description, specific aspects and embodiments in which this subject matter may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice this subject matter. Other embodiments may be utilized, and structural, logical, and electrical changes may be made without departing from the scope of this subject matter. References to “an,” “one,” or “various” embodiments in this disclosure do not necessarily refer to the same embodiment, and such references contemplate more than one embodiment. Therefore, the following detailed description should not be considered limiting, and its scope is defined only by the appended claims and the full scope of their legal equivalents.

[0036] Figure 1 A label placement inspection and quality control system 100, configured to evaluate label placement on each of a plurality of objects or samples 101, is illustrated by way of example, not limitation. System 100 can be implemented in a machine vision-based product inspection and quality control platform in a packaging and assembly facility. The objects or samples 101 are substantially similar in size, and each object or sample has a label affixed to the exterior of the device. The labels on the devices are substantially similar in size. By way of example, not limitation, the object or sample 101 can be a drug injection device, such as a syringe pre-filled with a drug or a self-injection pen (also known as an autoinjector). The drug injection device may include two interconnectable elements. The label may be displayed on the cylindrical exterior portion of the drug injection device.

[0037] System 100 may include an imaging device 110, a tag quality controller 120, a machine learning (ML) system 130, and a user interface 140. System 100 may optionally include a robotic system 150 capable of controlled manipulation of objects or samples that have detected defective, missing, or misaligned tags. Imaging device 110 can acquire an image of each object or sample in object or sample 101. In an example, imaging device 110 is a camera. In some examples, imaging device 110 includes one or more dedicated cameras or imaging sensors, such as photomultiplier tubes (PMTs), charge-coupled devices (CCDs), time-delay integration (TDI) cameras, linear photodiode arrays, or any other suitable imaging sensors known in the art.

[0038] In the example, system 100 includes only one standardized imaging device 110 to perform in-process image capture and analysis, metrological measurements, and label placement checks and defect detection. This single imaging device-system may be preferred in some applications to reduce spare parts inventory and features a common platform for all inspections. Other advantages of a single imaging device-system compared to a multi-camera system include lower requirements for expertise in operating and maintaining the imaging device and software platform, and cost savings. Alternatively, in some examples, system 100 may include more than one imaging device. These imaging devices may have different settings (e.g., zoom settings, illumination or lighting conditions, contrast settings, exposure levels, or viewing angles toward the object or labels on it) to obtain corresponding images of the object. Images obtained from multiple imaging devices can be analyzed for object identification, label inspection, or metrological measurements.

[0039] Tag quality controller 120 may include circuit groups comprising one or more other circuits or sub-circuits, such as image processor 122, metering circuit 126, and tag quality inspection circuit 128. Image processor 122 may further include object recognition circuit 123 and feature extraction circuit 124. The circuits or sub-circuits may perform the functions, methods, or techniques described herein individually or in combination. In the example, the hardware of the circuit group may be designed to perform specific operations (e.g., hard-wired). In the example, the hardware of the circuit group may include variable-connected physical components (e.g., signal processors, execution units, transistors, combinational logic, simple circuits, etc.), including a computer-readable medium that is physically modified (e.g., magnetically, electrically, optically movable to place invariant mass particles, etc.) to encode instructions for specific operations. When connecting physical components, the underlying electrical properties of the hardware components may, for example, be changed from an insulator to a conductor and vice versa. Instructions enable embedded hardware (e.g., an execution unit or loading mechanism) to create members of the circuit group in the hardware via variable connections to perform specific operations during operation. Therefore, when the device is operating, the computer-readable medium is communicatively coupled to other components of the circuit group members. In the example, any physical component can be used in more than one member of more than one circuit group. For example, in operation, the execution unit may be used at one point in time in a first circuit of a first circuit group and reused at a different time by a second circuit in the first circuit group or by a third circuit in the second circuit group. Therefore, the tag quality controller can be implemented in various combinations of hardware or software or firmware.

[0040] In various examples, some functionality of the label quality controller 120 may be implemented as part of a microprocessor circuit. The microprocessor circuit may be a dedicated processor, such as a digital signal processor, application-specific integrated circuit (ASIC), microcontroller, microprocessor, or other type of processor for processing information, including bodily activity information. Alternatively, the microprocessor circuit may be a general-purpose processor that can receive and execute a set of instructions for performing the methods or techniques described herein.

[0041] Object recognition circuit 123 can identify objects of interest from the background of an image. Object detection or image segmentation techniques can be used for object recognition. The object detection task is to predict one or more bounding boxes around a part of an object based on the pixel values ​​of an image to identify the location of the object or a part of it on the image. Object recognition circuit 123 can use edge detection or template matching techniques to determine the bounding boxes. For example, object recognition circuit 123 can generate multiple rectangular bounding boxes defined by the position of the top-left corner (two coordinates) and the width and height (in pixels). In contrast to object detection, the image segmentation task is to predict a “mask”, which is a pixelated outline of an object or a part of an object on a map. Through image segmentation, object recognition circuit 123 can generate a transformed image with the same dimensions as the original image, but for each pixel, it only has a Boolean value indicating whether the pixel belongs to the object of interest (e.g., a value “1” indicates an edge of the object) or does not exist (e.g., a value “0” indicates a non-edge of the object).

[0042] Feature extraction circuit 124 can extract one or more image features of the object and / or one or more image features of the label, provided that these image features are detected by object recognition circuit 123. Examples of extracted image features include statistical or morphological features of pixel values, such as corners, edges, blobs, curvature, accelerated robust features (SURF), or scale-invariant feature transform (SIFT) features. In the example where the object or sample 101 is a labeled drug injection device, the morphological features extracted from the image may include one or more edges of the object and / or one or more edges of the label. Edges can be detected by determining where the image intensity changes abruptly (e.g., exceeding a threshold) to produce discontinuities in the image. Examples of edge detection algorithms include Canny, Sobel, Prewitt, Roberts, Canny-Deriche, Laplacian of Gaussian, or zero-crossing. In some cases where the edges detected by the algorithm are too short, too small, or discontinuous (which makes it difficult to produce consistent and accurate distance measurements), the feature extraction circuit 124 can post-process the detected edges by interpolating and / or extrapolating edge pixels to produce continuous and smooth edges.

[0043] One or more of the object recognition circuit 123 or feature extraction circuit 124 can each use one or more trained machine learning (ML) models 134 to perform their respective object detection and feature extraction tasks. The ML models 134 can be trained using training module 131 and then stored in storage device 133. Training module 131 and storage device 133 can be part of an ML system 130 communicatively coupled to the label quality controller 120. In this example, the ML system 130 can be a remote computing and storage system, such as a cloud comprising one or more computing devices (e.g., servers) configured to provide cloud-based services, including, for example, data storage, computing services, and the provision of customer service. The label quality controller 120 or portions thereof (e.g., image processor 122) can remotely and securely access data and services, such as the trained ML models 134, from the machine learning system 130.

[0044] In the example, the trained ML models 133 may have a neural network architecture. The neural network architecture may include an input layer, one or more hidden layers, and an output layer. Images acquired from the imaging device 110 may optionally be fed, along with other auxiliary input data, into the input layer of the trained ML models 134, manipulated through one or more hidden layers, and output from the output layer. The trained ML models 134 enable the system 100 to perform various tasks (including object recognition and feature extraction) by making inferences based on patterns discovered in image analysis, without explicit programming. The trained ML models 134 explore the construction of algorithms (e.g., ML algorithms) that can learn from existing data and make predictions about new data. Such algorithms operate by constructing the trained ML models 134 from a training dataset to make data-driven predictions or decisions represented as outputs or evaluations.

[0045] Training module 131 can use a training dataset stored in device image database 132 to generate or update multiple ML models 134. Training dataset 132 may include images of other objects similar to the object being examined, such as images of drug injection devices of the same or similar type as the examined drug injection device. In addition to or as a substitute for images of “real” objects or samples, the training dataset may include artificially or algorithmically modified versions of object images. To enhance the robustness of the multiple trained ML models 134 in object and label recognition and feature extraction (e.g., edges of objects or labels), the training dataset may include images obtained under variations in imaging modes and conditions, object orientation, or label placement on the object. For example, the training dataset may include images of objects obtained under different zoom settings, lighting or illumination conditions, contrast settings, sharpness settings, exposure levels, or viewpoints toward the object or labels on it, or images of objects in different orientations (e.g., horizontal, vertical, or angular or rotated positions). Alternatively, the training dataset may include images of objects with missing labels, labels placed at different external locations on the object, labels of varying quality (e.g., new and old labels, rough labels, labels with worn or torn edges or corners), or any other observable features of the labels. In some examples, the training dataset may include images of objects that have been artificially or algorithmically modified to blur or partially occlude morphological features of the object (e.g., object edges or label edges).

[0046] Training module 131 can train (multiple) ML models using either supervised or unsupervised learning. Supervised learning uses prior knowledge (e.g., examples that associate inputs with outputs or results) to learn the relationship between inputs and outputs. The goal of supervised learning is to learn a function that best approximates the relationship between training inputs and outputs given some training data, so that the ML model can implement the same relationship given inputs to generate the corresponding output. When using supervised learning to train ML models, the training dataset can further include “desired” outputs, such as the user-identified location, label, and / or morphological features of objects. The “desired” outputs and associated images can be stored in the device image database 132. Unsupervised learning is the automated training of ML algorithms using deterministic information that does not require manual correction by the ML algorithm. Unsupervised learning is useful in exploratory analysis because it can automatically identify structures in the data.

[0047] Common tasks in supervised learning are classification and regression problems. Classification problems, also known as categorization problems, aim to classify items into one of several class values. Regression algorithms aim to quantify some items (e.g., by providing scores for some input values). Some examples of commonly used supervised-ML algorithms are logistic regression (LR), Naive Bayes, random forests (RF), neural networks (NN), deep learning, deep neural networks (DNN), matrix factorization, and support vector machines (SVM). Deep learning, or DNN, typically refers to a neural network consisting of multiple (e.g., more than three) hidden layers. Examples of DNNs include convolutional neural networks (CNN), recurrent neural networks (RNN), deep belief networks (DBN), or hybrid neural networks that include two or more neural network models of different types or configurations. Some common tasks in unsupervised learning include clustering, representation learning, and density estimation. Some examples of commonly used unsupervised learning algorithms are K-means clustering, principal component analysis, and autoencoders.

[0048] Another type of machine learning is federated learning (also known as collaborative learning), which trains algorithms across multiple distributed devices that store local data without exchanging data. This approach contrasts with traditional centralized machine learning techniques, where all local datasets are uploaded to a single server, and more classic distributed methods, which typically assume that local data samples are uniformly distributed. Federated learning enables multiple participants to build public, robust machine learning models without sharing data, thus allowing for the resolution of critical issues such as data privacy, data security, data access permissions, and access to heterogeneous data.

[0049] Training an ML model can be performed continuously or periodically, or in near real-time as additional object images become available. Training involves algorithmically adjusting one or more ML model parameters until the trained ML model meets a specified training convergence criterion, such as the difference between the model output (for a given input) and the "desired" output (e.g., the user-identified location or morphological features of an object) falling below a specified threshold.

[0050] In some examples, multiple ML models can be trained, validated, and used individually (during the inference phase) to achieve different objectives, such as detecting objects or labels on items, or detecting one or more morphological features, such as the edges of objects or the edges of labels. In the examples, a first ML model might be trained to recognize objects (e.g., using bounding boxes around the object), a second ML model might be trained to recognize labels on objects (e.g., using bounding boxes around the label), a third ML model (or a set of ML models) might be trained to detect one or more edges of the object, and a fourth ML model (or a set of ML models) might be trained to detect one or more edges of the label. In some examples, separate specialized ML models are each trained to detect the corresponding edge (object edge or label edge), such as a first trained ML model used to recognize the first edge, and a second trained ML model used to recognize the second edge. Individually trained ML models may differ in model architecture (e.g., the number of hidden layers, the number of neurons in any layer) or at least one model parameter (e.g., the weights applied to any neuron or the biases introduced in any layer). The following section discusses... Figure 3 Figure 5 illustrates an example of training and using an ML model to perform automated inspection and quality control checks on label placement on drug injection devices.

[0051] Metric circuit 126 can determine a spatial metric between the image features of the object generated by feature extraction circuit 124 and the image features of the label. The spatial metric can be used to locate the label on the object. Examples of spatial metrics can include distance, shape, area, nearest point on the object's edge, and other geometric features of the label. In an example, the spatial metric includes the distance (d) between the edge (A) of the object and the edge (B) of the label on the object. AB In the example, the detected edges can be represented by a set of image pixels with a spatial distribution. For example, the detected edge (B) of the label can have a width along the longitudinal direction of the cylindrical body of the device. To generate a consistent and reliable distance metric, the metering circuit 126 can determine the centroid of the image pixels as the representative location of the edge and measure the distance d between the corresponding representative locations of two edges. AB Additionally, in some examples, the distance range can be determined using the width of the spatial distribution of pixels at the edge, such as the minimum distance (d) between the nearest pixels of the corresponding edge.AB-min The maximum distance (d) between the farthest pixel interval of the corresponding edge and the edge. AB-max The distance range between ).

[0052] Based at least in part on spatial metrics, the label quality inspection circuit 128 can generate label placement indicators to indicate whether the label placement is within a predetermined tolerance on the object. For example, if the distance d between edges... AB Within a predetermined range, a positive placement indicator can be generated to indicate the appropriate label location, and optionally, along with meeting other criteria, to indicate passing a quality check of label placement. However, if the determined distance d AB If outside the predetermined range, it indicates that the label is placed incorrectly or misplaced, and the label placement has not passed the quality control inspection.

[0053] User interface 140 may include an output unit and an input unit. The output unit may include a display for showing a label placement indicator to the user. It may also display the original image, intermediate image processing results (such as detection of objects and labels, as well as edges or other morphological features), and metrological measurement results (such as distances between edges). In an example, user interface 140 may generate an alert when the label placement indicator indicates that a label does not meet external positioning requirements. The input unit of user interface 140 may receive user input for programming or adjusting parameters of components of system 100, such as the imaging mode and conditions of imaging device 110, or one or more operating parameters of image processor 122 for object recognition, feature extraction, or metrological measurement. In some examples, the input unit may receive auxiliary input (in addition to images of objects) from the user or from an external source or storage device that can be used to train multiple ML models. For example, the user may specify image features of interest, such as specific edges of objects or labels. Training module 131 may use stored images from device image database 132 and auxiliary input from the user to train multiple ML models to recognize features of interest from input images. Including auxiliary inputs during model training can improve the robustness and accuracy of the trained ML model in object recognition and feature detection. The label quality controller 120 can use (multiple) trained ML models to detect corresponding features of interest from images acquired by the imaging device 110 and perform label quality control checks.

[0054] The label placement indicator may be provided to the robot controller of the robot system 150, either separately or alternatively, to enable robot manipulation of objects according to the label placement indicator, such as picking up and placing misaligned objects via an electric robotic arm.

[0055] As described above, in some examples, the object or sample 101 being inspected is a labeled drug injection device, such as a syringe or self-injection pen. The drug injection device may include two interconnectable elements, such as the plunger and barrel of a syringe. The drug injection device may include a cylindrical device body displaying a device label on its outer surface. The label may include product information and usage information, including, for example, product name, stock unit (SKU) code, expiration date, active ingredient, indications, dosage and administration, contraindications, warnings and precautions, adverse reactions, drug interactions, etc. In device manufacturing or inspection, a portion of quality assurance is ensuring that the label is placed in the correct location on the exterior of the device body. Improper label placement may be considered a failure to pass quality control checks. As part of the quality control process, system 100 may automatically identify the drug injection device and its label from an image (e.g., by separating it from the background of the image using a bounding box in object detection or an object mask in image segmentation), inspect and perform metrological measurements of the label placement on the cylindrical exterior, and determine whether the label is placed in the desired location within acceptable tolerances on the drug injection device. This is by way of example, not limitation. Figure 2Images 200A, 200B, 200C, and 200D illustrate the image features of this drug injection device and the placement of a label on the positioning device. Each device includes a cylindrical device body 201 and an insertable element 210 configured to fit snugly within a tubular compartment of the cylindrical device body 201. The device images also show a label 220 placed on the outer portion of the cylindrical body 201, and a seam 212 between the two interconnectable elements when they are engaged with each other. Images 200A-200D were each obtained at correspondingly different angles or rotational positions of the drug injection device relative to the lens orientation of the imaging device 110, thus the label in each of the corresponding images 200A-200D has different observability. For example, while images 200A, 200B, and 200D each show a relatively large portion of the label 220, only a small portion of the label 220 appears in image 220C. The observability of seam 212 also varies with different device rotation positions, as shown in images 200A-200D. Besides the device's attitude or orientation, other factors can also affect the observability and detectability of the label. These factors include, for example, different imaging modes and conditions (e.g., zoom settings, lighting or illumination conditions, sharpness, exposure level, or viewing angle toward the object or the label on it), label location, and label quality (e.g., new and old labels, rough labels, labels with worn or torn edges or corners). Variations in label observability and detectability can make it difficult to accurately and consistently detect one or more of the features of interest in the image, such as the edges of seam 212, the edge 222 of label 220, or the edge 232 at the distal end of cylindrical body 201. For example, some edges may not have sufficient contrast with the background (e.g., due to poor lighting or illumination, or some device attitude or orientation). The location of some edges is inconsistent, so their length may not always extend the length of the entire predefined area. In some instances, nearby edges or elements on the exterior of the device may confuse the edge of interest.

[0056] As mentioned above Figure 1 The described ML-based object and edge detection provides robust edge detection and metrology, and improves the efficiency of label placement verification and quality assurance processes. Among other factors, the improved performance is achieved, at least, through training the ML model on a training dataset that includes rich variations in device images (e.g., various object rotations, poses, orientations, etc.). Figure 2(as shown), labels placed in different locations with different observability, variations in label quality (e.g., new and old labels, rough labels, labels with worn or torn edges or corners), variations in imaging modes and conditions (e.g., zoom settings, lighting or illumination conditions, sharpness, exposure levels, or viewing angle toward the object or the label on it), or artificial or algorithmic modifications or distortions to the image to blur or partially obscure the morphological features of interest.

[0057] In examples of using supervised learning to train (multiple) ML models, for any input image, the training dataset may include "desired" outputs, such as the desired location of objects, the presence or absence and location of labels, or features of interest for objects or labels (e.g., the distal end 232 of a cylindrical body, seam 212, and label edge 222). The "desired" outputs for the input image may be provided as user input and stored in the device image database 132. Figure 2 As demonstrated, for object and edge detection purposes, bounding boxes (or other identifiers) can be drawn around each object or morphological feature (e.g., edge) of interest. Once the ML model has been trained and meets the convergence criteria, it can be used to automatically detect objects, labels, and morphological features from new images of objects.

[0058] In some examples, auxiliary inputs can be provided to training module 131 to improve the robustness and accuracy of the trained ML model in object recognition and feature detection. The auxiliary inputs may include user-specified features of interest to be detected from the image. Figure 3 Image 300 illustrates, rather than limits, a drug injection device and a set of objects and label features that can be used to position labels on the device. Similar to Figure 2In the images shown, the drug injection device has two interconnectable elements, comprising a cylindrical outer body 301 and an insertable element 310 for assembly within a tubular compartment of the cylindrical body 301. The drug injection device includes a label 320 shown on the outer portion of the cylindrical body 301. In this example, the label 320 is represented by a black ablated area. Candidate features also shown in image 300 include a seam 312 between the two interconnectable elements when they engage with each other, an edge 322 of the label 320, and an edge 332 at the distal end of the cylindrical body 301. The location of the seam 312 can be used as a reference point for metrological measurements to determine the label's location. When the seam 312 is formed between the cylindrical body 301 and the insertable element 310 assembled within the body 301, the seam 312 can appear in the image with low contrast and therefore low observability and detectability. Additionally, the seam 312 may overlap or closely approximate other edges on the device body 301 (e.g., the edge of a nearby label), text, printed material, or structural elements, which can blur the edge of interest when using conventional intensity gradient-based edge detection methods. Edge 322 (shown in this example as the black ablated edge of label 320) is a feature directly related to the label's location. However, this feature may not always appear in the image because it is susceptible to device orientation, rotation, and pose during imaging. Device contour features (such as the edge 332 at the distal end of the cylindrical device body 301) have been widely used in existing machine vision-based device inspection, metrology, and quality control processes. However, since the distal end of the cylindrical device body 301 is typically not straight but slightly curved, simple gradient-based edge detection methods may not reliably detect sharp, straight edges and may produce inconsistent distance measurements between edges of interest.

[0059] The inventors have realized that, as Figure 3 As shown, conventional machine vision-based label placement inspection and quality control checks are disadvantageous, particularly with the use of error-prone features (such as the edge 332 of the distal portion of the cylindrical device body 301) and simple gradient-based edge detection methods, which are susceptible to interference from nearby uninterested edges or image features, variations in imaging patterns and conditions, or device pose and orientation. Various embodiments described in this document provide improved solutions, including, for example, avoiding the device end edge 332 and instead using other features such as seam 312 and / or label edge 322. Users can pre-select features of interest via input units of the user interface 140. Figure 4 Examples of features of interest selected by the user for identification by the label placement inspection and quality control check system 100 are shown. In these examples, the features of interest include those related to… Figure 3The seam and label edge on image 400, which shows a drug injection device similar to the one shown. The user mark 412 for the seam and the user mark 422 for the label edge may be discontinuous, such as... Figure 4 As shown. Given the variable nature of device orientation, pose, and rotation, the features of interest (e.g., edges) are not entirely consistent across different images. However, when utilizing elements containing those described above... Figure 1 and Figure 2 When a sufficient number of images describing the imaging patterns and conditions, object orientation, or label placement variations on the object are used to train the ML model, the features of interest can be appropriately identified and located. The metrology circuit 126 can determine the distance between a detected seam and a detected label edge, and the label quality inspection circuit 128 can determine a label placement indicator based on whether the determined distance is within a predetermined tolerance.

[0060] Figures 5A to 5B It is a diagram that shows an example of training an ML model and using the trained ML model to identify objects or labels from an image and detect features of interest, and optionally make a pass / fail quality control check decision. Figure 5A This demonstrates the training phase of an ML model, during which objects or samples (such as...) can be used. Figures 2 to 4 The training dataset of multiple images 510 of the drug injection device shown is used to train the ML model 534 (an example of the trained ML model 134). The objects or samples being imaged have various rotations, poses, or orientations. Each object or sample has a label that can be placed at different locations outside the device, or labels with different label qualities (e.g., new and old labels, rough labels, labels with worn or torn edges or corners). Images of the objects or samples can be acquired under various imaging modes and conditions (e.g., zoom settings, lighting or illumination conditions, sharpness, exposure levels, or viewing angle toward the object or the label on it). In some examples, images can be manually or algorithmically modified to blur or partially occlude morphological features of interest. Each image in the training dataset 510 may include a user identifier of the object to be identified and its label, such as... Figure 3 The cylindrical device body 301 and label 320 shown, as well as the markings indicating the features of interest, such as Figure 4The seam 312 and label edge 322 are shown. The ML model 534 can be trained by the training module 131 using supervised learning, unsupervised learning, or reinforcement learning. Examples of ML model architectures and algorithms can include, for example, decision trees, neural networks, support vector machines, or deep learning networks. Examples of deep learning networks include convolutional neural networks (CNNs), recurrent neural networks (RNNs), deep belief networks (DBNs), or hybrid neural networks comprising two or more neural network models of different types or different model configurations. In some examples, when the edges detected by the algorithm are too short, too small, or discontinuous (which makes distance measurements difficult or inaccurate), the ML model 534 can be trained to use interpolation and / or extrapolation of edge pixels to produce continuous and smooth edges.

[0061] Training of the ML model can be performed continuously or periodically, or near real-time when additional image data is available. The training process involves algorithmically adjusting one or more ML model parameters until the trained ML model meets a specified training convergence criterion. The trained ML model 534 can establish a correspondence between an input image of an object (e.g., a drug injection device) and the recognition, features, or interest in the object or label, or a pass / fail decision regarding the placement of the label on the examined object.

[0062] Figure 5B A reasoning phase is illustrated, in which an imaging device 110 is used during the manufacturing or inspection process to acquire a live image 520 of the object and apply it to a trained ML model 534 for automated and real-time quality control checks of label placement. The trained ML model 534 can identify objects and labels, detect the location of edges of interest, and present the results to a user. In the example, the output may include an image 540 with visual identifiers of the edges detected by the algorithm, such as a highlighting of the location of the seam 512 and label edge 522 superimposed on the input live image 520. In some examples, certain operations and decision-making processes implemented in the metering circuit 126 and / or the label quality inspection circuit 128 may alternatively be implemented during the model training process, such that the trained ML model 534 may include another layer or additional output neurons (in a neural network model) that produce additional outputs 550, such as the distance (and optionally a range of distances) between the seam 512 and label edge 522 detected from the input image 520, or a pass / fail decision regarding label placement. The output can be transmitted to a human operator or provided to a robotic system to facilitate robot-assisted picking and placing of the inspected object.

[0063] Figure 6This is a flowchart illustrating an example method 600 for evaluating label placement on an object. Method 600 can be implemented using a label placement inspection and quality control check system 100. Although the processes of method 800 are drawn in a flowchart, they do not need to be performed in a specific order. In various examples, some processes may be performed in a different order than that shown herein.

[0064] At point 610, an image of the object or sample to be inspected for quality control checks can be obtained, for example, using imaging device 110. This object or sample is one of several objects or samples of substantially similar size, each with a label of substantially similar size. An example of the object or sample to be inspected is a drug injection device, such as... Figures 2 to 4 The syringe or self-injection pen shown. This drug injection device may include two interconnectable elements and a cylindrical outer portion for displaying a label.

[0065] At 620, at least one trained machine learning (ML) model that has been created and stored in a storage device can be received. This at least one ML model establishes a correspondence between an input image and objects identifiable from the input image, labels associated with the objects, and features of the objects or labels. The at least one ML model can be trained using a training dataset, which may include images of other objects similar to the object or sample being examined, such as images of drug injection devices of the same or similar type as the drug injection device being examined. In addition to or as a substitute for “actual” images of objects obtained by an imaging device (such as imaging device 110), the training dataset may include artificially or algorithmically modified versions of “actual” images. The training dataset may include images obtained under variations in imaging modes and conditions, object orientation, or label placement on the object. For example, the training dataset may include images of objects obtained under different zoom settings, lighting or illumination conditions, contrast settings, sharpness settings, exposure levels, or viewing angles toward the object or its labels, or images of objects placed in different orientations. Alternatively, the training dataset may include images of objects with missing labels, labels placed at different external locations on the object, labels of varying quality (e.g., new and old labels, rough labels, labels with worn or torn edges or corners), or varying observability. In some examples, the training dataset may include object images that have been manually or algorithmically modified to blur or partially occlude the morphological features of the object (e.g., object edges or label edges). Training the ML model with highly varied images can enhance the robustness of the trained ML model(s)134 in recognizing objects or labels and extracting features from them.

[0066] Training an ML model can be performed continuously or periodically, or in near real-time as additional object images become available. Training involves algorithmically tuning one or more ML model parameters until the trained ML model meets a specified training convergence criterion, such as the difference between the model output (for a given input) and the "desired" output.

[0067] At 630, at least one trained ML model received at step 610 can be applied to the image of the object obtained from step 610 to identify at least one morphological feature of the object and at least one morphological feature of the label. In the example, multiple ML models can be trained individually. For example, a first ML model can be trained to identify objects (e.g., one or more bounding boxes surrounding the object), a second ML model can be trained to identify labels on the object (e.g., one or more bounding boxes surrounding the label), a third ML model (or a set of ML models) can be trained to detect one or more edges of the object, and a fourth ML model (or a set of ML models) can be trained to detect one or more edges of the label. In some examples, separate specialized ML models are each trained to detect the corresponding edge (object edge or label edge), such as a first trained ML model used to identify a first edge and a second trained ML model used to identify a second edge. The individually trained ML models described above may differ in model architecture or at least one model parameter.

[0068] At position 640, a spatial metric can be determined between the morphological features of the identified tag and the morphological features of the identified object. Examples of spatial metrics can include distance, shape, area, nearest points on the object's edges, and other geometric features of the tag. In the example, the spatial metric includes the distance between the edge of the object and the edge of the tag on the object. In the example, the detected edges can be represented by a set of image pixels with a spatial distribution. The centroids of the image pixels can be determined as representative locations of the edges, and the distance between corresponding representative locations of two edges can be measured. Additionally, in some examples, the spatial distribution of the edge pixels can be used to determine the distance range, such as the range between the minimum distance between the nearest pixels of the corresponding edges and the maximum distance between the farthest pixels of the corresponding edges.

[0069] In the example where the object being examined is a drug injection device, one or more specialized, trained ML models can be used to identify the seam between the two interconnectable elements of the injection device when they are engaged with each other, as well as the edge of the label, from an image of the object. The distance between the detected seam and the detected label edge can be measured, as described above. Figure 3 and Figure 5B As described.

[0070] At 650, based at least in part on the determined spatial measurements, a label placement indicator can be generated to indicate whether the label is placed within a predetermined tolerance on the object. For example, if the distance between the first and second edges is within a predetermined range, it indicates affirmative placement or an appropriate label position. However, if the distance between the first and second edges is outside the predetermined range, it indicates negative placement or label misalignment. An alert can be provided to the user when the label placement indicator indicates that the label has not met the external positioning requirements. The label placement indicator can be additionally or alternatively provided to the robot controller of the robotic system to facilitate robotic manipulation of the object according to the label placement indicator, such as picking up and placing misaligned objects via an electric robotic arm.

[0071] Figure 7 A block diagram of an example machine 700 is shown in general, on which any or more of the techniques (e.g., methods) discussed herein can be performed. Various parts of this specification can be applied to the computational framework of various parts of a label placement inspection and quality control inspection system 100.

[0072] In alternative examples, machine 700 may operate as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, machine 700 may operate as a server machine, a client machine, or both in a server-client network environment. In the examples, machine 700 may act as a peer-to-peer (P2P) (or other distributed) network environment. Machine 700 may be a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), mobile phone, web appliance, network router, switch or bridge, or any machine capable of (sequentially or otherwise) executing instructions specifying actions to be taken by that machine. Further, while only a single machine is shown, the term "machine" should also be considered as any collection of machines that individually or jointly execute a set (or more) of instructions to perform any or more of the methods discussed herein, such as cloud computing, Software as a Service (SaaS), and other computer cluster configurations.

[0073] As described herein, examples may include logic or multiple components or mechanisms, or may be operated by logic or multiple components or mechanisms. A circuit group is a collection of circuits implemented in a tangible entity including hardware (e.g., simple circuits, gates, logic, etc.). The membership of a circuit group can change flexibly over time and with the variability of the underlying hardware. A circuit group includes members that can perform a specified operation individually or in combination during operation. In the examples, the hardware of the circuit group may be designed invariably to perform a specific operation (e.g., hard-wired operation). In the examples, the hardware of the circuit group may include physically connected components (e.g., execution units, transistors, simple circuits, etc.), including computer-readable media that are physically modified (e.g., magnetically or electrically movable to place invariant mass particles, etc.) to encode instructions for a specific operation. When connecting physical components, the underlying electrical properties of the hardware components change, for example, from insulator to conductor, and vice versa. Instructions enable embedded hardware (e.g., execution units or loading mechanisms) to create members of the circuit group in the hardware via variable connections to perform specific operations during operation. Therefore, when the device is operating, the computer-readable medium is communicatively coupled to other components of the circuit group members. In the example, any physical component can be used in more than one member of more than one circuit group. For example, during operation, an execution unit may be used at one point in time in a first circuit of a first circuit group and reused at a different time by a second circuit in the first circuit group or by a third circuit in the second circuit group.

[0074] Machine (e.g., computer system) 700 may include a hardware processor 702 (e.g., a central processing unit (CPU), graphics processing unit (GPU), hardware processor core, algorithm-specific ASIC, or any combination thereof), main memory 704, and static memory 706, some or all of which may communicate with each other via an interconnect link (e.g., bus) 708. Machine 700 may further include a display unit 710 (e.g., raster display, vector display, holographic display, etc.), an alphanumeric input device 712 (e.g., keyboard), and a user interface (UI) navigation device 714 (e.g., mouse). In the example, display unit 710, input device 712, and UI navigation device 714 may be a touch screen display. Machine 700 may additionally include a storage device (e.g., drive unit) 716, a signal generation device 718 (e.g., speaker), a network interface device 720, and one or more sensors 721 (e.g., a global positioning system (GPS) sensor, compass, accelerometer, or other sensor). Machine 700 may include output controller 728, such as a serial connection (e.g., Universal Serial Bus (USB)), a parallel connection, or other wired or wireless (e.g., infrared (IR), near field communication (NFC)) connection, to communicate with or control one or more peripheral devices (e.g., printers, card readers, etc.).

[0075] Storage device 716 may include machine-readable medium 722 on which one or more sets of data structures or instructions 724 (e.g., software) are stored, which embody or be utilized by any one or more of the techniques or functions described herein. During execution by machine 700, the instructions 724 may also reside wholly or at least partially within main memory 704, static memory 706, or hardware processor 702. In this example, one or any combination of hardware processor 702, main memory 704, static memory 706, or storage device 716 may constitute the machine-readable medium.

[0076] Although the machine-readable medium 722 is shown as a single medium, the term "machine-readable medium" can include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) configured to store one or more instructions 724.

[0077] The term "machine-readable medium" can include any medium capable of storing, encoding, or carrying instructions executable by machine 700 and enabling machine 700 to perform any one or more of the technologies disclosed herein, or any medium capable of storing, encoding, or carrying data structures used by or associated with such instructions. Examples of non-limiting machine-readable media can include solid-state memory as well as optical and magnetic media. In examples, mass-capacity machine-readable media includes machine-readable media with a plurality of particles having constant mass. Therefore, mass-capacity machine-readable media are not transient propagation signals. Specific examples of mass-capacity machine-readable media can include: non-volatile memory, such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EPSOM)) and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0078] The instructions 724 can be transmitted or received via network interface device 720, using a transmission medium, and through communication network 726, using any of several transport protocols (e.g., Frame Relay, Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), etc.). Example communication networks may include local area networks (LANs), wide area networks (WANs), packet data networks (e.g., the Internet), mobile phone networks (e.g., cellular networks), conventional telephone (POTS) networks, and wireless data networks (e.g., the IEEE 802.11 standard series known as WiFi®, the IEEE 802.16 standard series known as WiMax®), the IEEE 802.15.4 standard series, peer-to-peer (P2P) networks, etc. In the example, network interface device 720 may include one or more physical jacks (e.g., Ethernet, coaxial, or telephone jacks) or one or more antennas for connection to communication network 726. In the example, network interface device 720 may include multiple antennas to perform wireless communication using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) technologies. The term "transmission medium" should be considered to include any intangible medium capable of storing, encoding, or carrying instructions executable by machine 700, and includes digital or analog communication signals or other intangible media used to facilitate communication of such software.

[0079] The accompanying figures above illustrate various examples. One or more features from one or more of these examples can be combined to form other examples.

[0080] The methods described herein can be implemented, at least in part, by a machine or computer. Some examples may include a computer-readable or machine-readable medium encoded with instructions operable to configure an electronic device or system to perform the methods described in the examples above. Implementations of this method may include code, such as microcode, assembly language code, high-level language code, etc. Such code may include computer-readable instructions for performing various methods. The code may form part of a computer program product. Further, the code may be tangibly stored on one or more volatile or non-volatile computer-readable media during execution or at other times.

[0081] The detailed description above is intended to be illustrative and not restrictive. Therefore, the scope of this disclosure should be determined by referring to the appended claims and the full scope of their equivalents.

Claims

1. A system for evaluating the placement of a label on one of a plurality of objects, each object being substantially similar in size and having a label of substantially similar size, the system comprising: An imaging device configured to acquire an image of the object; A storage device configured to store at least one trained machine learning (ML) model trained to establish a correspondence between an input image and the recognition of an object from the input image, a label associated with the object, and features of the object or the label. as well as Programmable circuit, configured as follows: The at least one trained ML model is applied to the obtained image to identify the morphological features of the object and the morphological features of the label, and to determine the spatial metric between the identified morphological features of the label and the identified morphological features of the object. as well as A label placement indicator is generated, at least in part, based on the determined spatial metric, to indicate whether the label placement is within a predetermined tolerance on the object.

2. The system as claimed in claim 1, wherein, The morphological features of the object include its first edge, the morphological features of the label include its first edge, and the spatial metric includes the distance between the first edge of the object and the first edge of the label. The programmable circuit is configured to generate the label placement indicator, which includes one or more of the following: (i) a positive placement indicator if the distance between the determined first edge and the second edge is within a predetermined range; or (ii) a misalignment indicator if the distance between the determined first edge and the second edge is outside the predetermined range.

3. The system as described in claim 2, wherein, The object is a drug injection device, which includes two interconnectable elements and a cylindrical outer portion for displaying the label. The object's morphological features include its seams.

4. The system as described in claim 3, wherein, The first edge and the second edge are each represented by a corresponding set of image pixels spatially distributed at least along the longitudinal direction of the outer portion of the cylinder. The programmable circuit is configured to determine the spatial metric, which includes the distance between the centroids of the respective image pixel sets representing the first edge and the second edge.

5. The system as described in claim 2, wherein, The at least one trained ML model includes a first trained ML model and a second trained ML model, respectively, for the first edge and the second edge. The programmable circuit is configured to apply the first trained ML model to the acquired image to identify the first edge, and to apply the second trained ML model to the acquired image to identify the second edge.

6. The system of claim 5, wherein, The first trained ML model and the second trained ML model differ in at least one of the model architectures or model parameters.

7. The system as claimed in claim 5, wherein, At least one of the first trained ML model or the second trained ML model is a trained deep learning model.

8. The system of claim 1, wherein, The programmable circuit includes a training module configured to generate or update the ML model using a training dataset comprising multiple images of objects of substantially similar size, each having labels of substantially similar size and variations in the placement of the labels on the objects.

9. The system of claim 8, wherein, The training module is configured to further use the user-identified features of each of the multiple images in the training dataset to generate or update the ML model.

10. The system of claim 8, wherein, These multiple images include images taken under different imaging modes or conditions.

11. The system of claim 10, wherein, Imaging modes or conditions include one or more of the following: zoom settings, lighting or illumination conditions, contrast settings, sharpness settings, exposure levels, or viewing angles toward the object or label.

12. The system of claim 8, wherein, The multiple images include images of objects in different poses, orientations, or rotations.

13. The system of claim 8, wherein, The multiple images include those that have been manually or algorithmically modified to blur or partially obscure the object's morphological features.

14. The system of claim 1, wherein, The at least one trained ML model includes an object detection model that is trained to generate at least one bounding box from the input image that identifies the location of the object or the location of the label.

15. The system of claim 1, wherein, The at least one trained ML model includes an image segmentation model trained to determine the pixelated outline of the object or the pixelated outline of the label from the input image.

16. The system of claim 1, further comprising a user interface configured to generate a label misalignment alarm when the label placement indicator indicates that the label is not within a predetermined tolerance on the object.

17. The system of claim 1, further comprising a robot controller configured to manipulate the object by a robot in accordance with the label placement indicator.

18. A method for evaluating the placement of a label on one of a plurality of objects, each object being substantially similar in size and having labels of substantially similar size, the method comprising: Obtain an image of the object; Receive at least one trained machine learning (ML) model from the storage device, the ML model being trained to establish a correspondence between an input image and the recognition of an object from the input image, a label associated with the object, and features of the object or the label; The at least one trained ML model is applied to the obtained image of the object via a programming circuit to identify the morphological features of the object and the morphological features of the label. Determine the spatial measure between the morphological features of the identified tag and the morphological features of the identified object; as well as A label placement indicator is generated, at least in part, based on the determined spatial metric, to indicate whether the label placement is within a predetermined tolerance on the object.

19. The method of claim 18, wherein: The object's morphological features include its first edge; The morphological features of the label include its first edge; The spatial metric includes the distance between the first edge of the object and the first edge of the label; and The label placement indicator includes one or more of the following: a positive placement indicator if the distance between the determined first edge and the second edge is within a predetermined range; or a misalignment indicator if the distance between the determined first edge and the second edge is outside the predetermined range.

20. The method of claim 18, wherein, The object is a drug injection device, which includes two interconnectable elements and a cylindrical outer portion for displaying the label. The object's morphological features include its seams.

21. The method of claim 19, wherein, The at least one trained ML model includes a first trained ML model and a second trained ML model that differs from the first ML model in at least one of the model architecture or model parameters. The method includes applying the first trained ML model to the acquired image to identify the first edge, and applying the second trained ML model to the acquired image to identify the second edge.

22. The method of claim 18, further comprising generating or updating the ML model via an ML training module using a training dataset, the training dataset comprising multiple images of objects of substantially similar size, each of which has labels of substantially similar size and variations in the placement of the labels on the objects.

23. The method of claim 22, wherein, Generating or updating the ML model further includes using the user identification features of each of the multiple images in the training dataset.

24. The method of claim 22, wherein, These multiple images include images taken under different imaging modes or conditions, or images of objects in different orientations, poses, or rotations.

25. The method of claim 22, wherein, The multiple images include those that have been manually or algorithmically modified to blur or partially obscure the object's morphological features.

26. The method of claim 18, wherein, The at least one trained ML model includes an object detection model that is trained to generate at least one bounding box from the input image that identifies the location of the object or the location of the label.

27. The method of claim 18, wherein, The at least one trained ML model includes an image segmentation model trained to determine the pixelated outline of the object or the pixelated outline of the label from the input image.

28. The method of claim 18, further comprising generating a label misalignment alarm when the label placement indicator indicates that the label is not within a predetermined tolerance on the object.

29. The method of claim 18, further comprising generating control signals for manipulating the object by a robot according to the label placement indicator.

Images