Robotic system for performing pattern recognition-based inspection of pharmaceutical containers

Abstract

To address challenging inspection tasks by performing the inspection of pharmaceutical containers and / or the inspection of samples within such containers.SOLUTION: A robotic inspection platform comprises a robotic arm, an imager, and a controller. The controller causes the robotic arm to use its end effector to retrieve a container, and to manipulate the container such that the container is sequentially placed in a plurality of orientations while in view of the imager. The controller also causes the imager to capture images, with each of the images being captured while the container is in a respective one of the orientations. The controller also determines one or more attributes of the container, and / or a sample within the container, by analyzing the images using a pattern recognition model and, based on the attributes, determines whether the container and / or sample satisfies one or more criteria. If the container and / or sample fails to satisfy the criteria, the controller causes the robotic arm to place the container in an area reserved for rejected containers and / or samples.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] This application generally relates to the inspection of pharmaceutical containers such as syringes or vials, and / or the inspection of samples within such containers.

Background Art

[0002] In certain situations, such as quality control procedures for manufactured pharmaceuticals, it is necessary to examine fluid or other (e.g., lyophilized) samples for the presence of various attributes such as particles, bubbles, or contaminants. Undesirable particles or other attributes can originate from many different causes, such as the environment, incorrect handling or storage, or by-products / residues of formation, packaging, or filling. Depending on the situation, certain types of attributes (e.g., bubbles) may be acceptable, while other attributes (e.g., particles, certain types of particles, particles exceeding a certain size and / or number, etc.) may not be acceptable. If an unacceptable attribute is detected in a fluid sample, the fluid sample is rejected.

[0003] Image processing methods conventionally used for defect detection in large commercial inspection machines often struggle to achieve high accuracy when faced with difficult inspection tasks. This is particularly true for inspections that require evaluating samples with very similar appearances while defective and non-defective attributes can coexist. For example, in the inspection of liquid-filled syringes, it may involve distinguishing particles or dirt (defects) on the piston dome from harmless bubbles (non-defects) on the piston dome. In an automated process, this makes it difficult to ensure that defective samples are rejected at an acceptable rate (e.g., with a 100% probability or at least a 99.99999% probability, etc.), and it may also make it difficult to ensure that qualified samples are approved at an acceptable rate (i.e., not wrongly rejected). The former situation can potentially lead to dangerous conditions, and the latter situation can lead to high costs due to unnecessary disposal of qualified products. The poor accuracy of an automated system can sometimes be compensated for by other means (e.g., repeating the inspection, manual inspection, etc.), but such methods are generally expensive in terms of both time and cost. Similar drawbacks exist in relation to other quality procedures, such as when inspecting defects in the container itself. SUMMARY OF THE INVENTION MEANS FOR SOLVING THE PROBLEM

[0004] The embodiments described herein relate to systems that improve conventional inspection techniques. In particular, a robotic inspection platform uses pattern recognition (e.g., deep learning) models / methods to inspect pharmaceutical containers (e.g., syringes, vials, cartridges, etc.) and / or samples (e.g., fluids, lyophilized products, etc.) within such containers and can also handle very difficult inspection tasks. The pattern recognition model may be capable of reliably discriminating, for example, between particles and bubbles in a fluid sample or between specific types of particles and specific types of bubbles, etc. The robotic inspection platform may also be configured to facilitate the initial (pre-production) development or training of the pattern recognition model. In some embodiments and / or situations, the platform may be used to perform a primary (e.g., single or initial) inspection of the container and / or sample. Alternatively, the platform may be used to reinspect containers and / or samples that have already been identified as non-conforming or likely to be non-conforming by another automated inspection system (e.g., a conventional automated inspection system with lower reliability / accuracy).

[0005] In embodiments where the platform implements a machine learning model, the model may be trained using a large dataset of training images and a supervised learning method. The platform may be used under controlled conditions (e.g., accurate and reproducible container orientation and lighting conditions) to capture the training images, and then a human analyst may manually label the attributes of the training images. For example, the analyst may label / classify the objects within the training images as specific object types. To capture the training images, the robotic arm of the platform may sequentially pick up each of several containers, present the containers in front of the imager, and manipulate (e.g., rotate) the containers while images at different angles / orientations are being acquired. During the training process, containers having intentionally introduced defects (and / or holding samples having intentionally introduced defects) may be used to ensure that a human analyst encounters a sufficiently wide range of defects and / or types of defects when performing the classification task.

[0006] Once the model is trained and incorporated, the platform can switch to a "production", "runtime", "operation", or "processing" mode. In this mode, the robotic arm removes new containers (which may hold samples), just as it did during the training process, and manipulates the containers in front of the imager. Images for each container are stored in memory and can be processed by a computer using the trained pattern recognition model. The computer may also process the output of the model by applying one or more criteria to determine whether the container is "acceptable" or "unacceptable". If the container is unacceptable (due to a defective sample within the container and / or a defective container itself), the robotic arm may place the container in a box or other area specifically reserved for that purpose. The robotic inspection platform may be configured to inspect only the containers, only the product samples (e.g., the fluid or lyophilized product within the container), or both the containers and the samples, depending on the embodiment and / or circumstances. In one embodiment, as long as an appropriate machine learning model is introduced, for example, a single robotic inspection platform can perform any of these functions.

[0007] Those skilled in the art will understand that the figures included herein are for illustrative purposes and do not limit the present disclosure. The drawings are not necessarily to scale; instead, emphasis is placed on showing the principles of the present disclosure. In some cases, various aspects of the described embodiments may be shown exaggerated or enlarged to facilitate understanding of the described embodiments. In the drawings, like reference numerals in the various figures generally refer to components that are functionally and / or structurally similar.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

[0009] The various concepts introduced above and described in more detail below can be implemented in any of a number of manners, and the concepts described are not limited to any particular implementation manner. Examples of embodiments are provided for illustrative purposes.

[0010] FIG. 1 shows a robotic inspection platform 100 according to an embodiment of the present disclosure. The robotic inspection platform 100 includes a plate 102 that houses a number of containers 104. The containers 104 may be fully or partially transparent or translucent pharmaceutical containers such as syringes, vials, cartridges, etc., each of which may contain a sample (e.g., a fluid or a lyophilized medical product). Alternatively, in some embodiments and / or situations (e.g., when the robotic inspection platform 100 analyzes only the containers and not the samples within them), the containers 104 may be opaque. Each container 104 may be configured to receive any suitable amount of sample depending on the size and configuration of the plate 102. For example, each container 104 may be able to hold a few tens of nanoliters of fluid, or a few milliliters of fluid, etc. The plate 102 may include a number of cavities or wells (e.g., 80 wells, 96 wells, etc.) sized to securely but removably hold the containers 104.

[0011] The robot inspection platform 100 also includes an imaging system 106. The imaging system 106 is generally configured to illuminate a container such as the container 104 and capture an image of the illuminated container. For this purpose, the imaging system 106 may include a light source, an imager, and, if possible, one or more mirrors and / or other optical elements for redirecting the light from the light source and / or the light scattered by the sample (and / or the container itself) in a suitable manner. The light source may be a light emitting diode (LED) light or any other suitable type of light source, may provide light of any suitable wavelength (or range of wavelengths), and may include lights at one or more positions. The light source may have only fixed or manually adjustable parameters, or may have one or more computer-controlled parameters (e.g., intensity, direction, etc.). Although FIG. 1 shows only a single imager and one mirror, alternatively, the imaging system 106 may include multiple imagers arranged at different positions, multiple mirrors (or no mirrors), etc. The imager may include any suitable combination of hardware and / or software such as an image sensor, an optical stabilizer, an image buffer, a frame buffer, a frame grabber, a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS) device, etc. In some embodiments, the imager may include an imaging device operating at an invisible wavelength such as one or more infrared camera devices. Further, in some embodiments, the light source may include several lights having different configurations, orientations, parameters, etc., and the light source may be configured such that different lights can strobe. In this way, different images corresponding to different lights can be obtained for each image. Different strobe lights may be more suitable, for example, for revealing different attributes of the container and / or the sample.

[0012] The robot inspection platform 100 also includes a robot arm 110. The robot arm 110 has a plurality of articulated segments (e.g., seven segments capable of six - degree - of - freedom movement) and an end effector 112. Joints connect the various segments, and the segments can be articulated using controllable stepping motors or other suitable means. The robot arm 110 may incorporate motion sensors and / or other sensors to provide control feedback, ensuring that the robot arm 110 moves in the intended state within a certain tolerance range.

[0013] Figure 1 shows the end effector 112 holding a container (e.g., a syringe) 114 that the robot arm 110 has removed from within the container 104 in the plate 102. The end effector 112 has a hardness suitable for firmly fixing (e.g., gripping) the container and includes a material suitable for avoiding damage to a fragile (e.g., glass) container. The end effector 112 may operate by compressing the container 114 or by inserting an element into the container 114 and expanding it outward. The end effector 112 may incorporate one or more pressure sensors to provide control feedback, ensuring that the end effector 112 provides the intended amount of force within a certain tolerance range.

[0014] The robot arm 110 may include a motor or other suitable means for rotational operation / movement and / or translational operation / movement of an object held by the end effector 112. For example, the robot arm 110 may be able to adjust the roll angle, pitch angle, and yaw angle of the container 114 by rotating and / or tilting (for roll and pitch) the end effector 112 and / or moving the articulated arm segments (for yaw). In some embodiments, the robot arm 110 is a commercially available device such as the UR3 robot arm provided by Universal Robots (UR (registered trademark)).

[0015] The robot inspection platform 100 also includes a reject bin 120 for discarding defective containers and / or samples. In the embodiment of FIG. 1, the reject bin 120 is a simple container having four side walls, a bottom / floor, and an open top. In other embodiments, the reject bin 120 may be constructed differently (e.g., the container may include a top having an open slot, or a plate having wells similar to plate 102, etc.).

[0016] Although not shown in FIG. 1, the controller is communicatively coupled to at least a portion of the imaging system 106 and the robot arm 110. The controller may be a general-purpose computing device specifically programmed according to the embodiments described herein, or a dedicated controller device (e.g., a dedicated processor integrated within the imaging system 106 and / or the robot arm 110). An embodiment of the controller will be described in more detail below with reference to FIG. 3.

[0017] The controller coordinates the operation of the imaging system 106 and the robot arm 110 during runtime operation. As used herein, the term "runtime" operation refers to the operation of the robot inspection platform (e.g., platform 100) during production, for example, to apply quality control procedures after the manufacture and packaging of pharmaceutical containers and / or samples and before sale or other distribution. In some embodiments, as described in more detail below, the controller also coordinates the operation of the imaging system 106 and the robot arm 110 during the training of the pattern recognition model prior to runtime operation.

[0018] During runtime operation, the controller generates command signals and transmits the command signals to the imaging system 106 and / or the robot arm 110. The transmission of the command / control signals will be further described below with reference to FIG. 3.

[0019] Here, the control / operation of the robot inspection platform 100 according to an embodiment will be described. First, the controller causes the robot arm 110 to take out the container 114 from the plate 102 by fixing (e.g., gripping, sucking, etc.) the container 114 using the end effector 112, and moves the container 114 to a position where the container 114 can be clearly "seen" by the imager of the imaging system 106 (e.g., as shown in FIG. 1). The light source of the imaging system 106 illuminates the container 114 and (if any) the sample inside the container 114. The various components of the imaging system 106 (e.g., imager, mirror, light source) are configured such that the light from the light source passes through the container 114, is reflected or refracted or blocked by the objects (e.g., particles, bubbles, etc.) inside the container 114, and is incident on the imager. Additionally or alternatively (e.g., when the container 114 does not contain a sample), the light can be reflected or refracted or blocked by various parts of the container 114. For example, the light source may be in the "on" state for a long time, or the controller may turn on the light source in a timely manner.

[0020] Thereafter, the controller causes the imager to capture at least one image of the container 114 and, if possible, also the sample within the container 114. However, in some embodiments and / or applications where the container 114 holds a sample, the controller first causes the robotic arm 110 to agitate (e.g., vibrate, invert, etc.) the container 114 immediately prior to imaging. After the first image is captured, the controller causes the end effector 112 of the robotic arm 110 to manipulate the container 114 such that the container 114 is positioned in a series of different orientations. For example, the container 114 may be rotated in 20-degree increments about its longitudinal axis (e.g., providing a total of 18 different orientations) and / or tilted at various angles with respect to the vertical position shown in FIG. 1. This orientation may correspond exactly to the orientation used, for example, to obtain training images manually labeled by a human analyst as further described below. In each orientation, the controller may cause the imager to capture one or more images of the container 114 and the fluid sample within the container 114. That is, the controller may synchronize the imaging system 106 and the robotic arm 110 using command signals in a timely manner. In some embodiments, different lights of the light source of the imaging system 106 (e.g., lights having different configurations, orientations, characteristics, etc.) strobe for each orientation of the container 114, and at least one image of the container 114 is captured for each light before moving the container 114 to the next orientation. Alternatively, the robotic arm 110 may move the container 114 through all orientations with respect to a first light and then move the container 114 through all orientations with respect to a second light, etc.

[0021] Thereafter, the controller may process the captured image and determine whether the container 114 (e.g., the container itself or the sample within the container 114) should be rejected. If it is rejected, the robotic arm 110 discards the container 114 by placing it in the reject bin 120. If it is not rejected, the robotic arm 110 may return the container 114 to the well of the plate 102 from which the container 114 was removed, or place it in another appropriate location. The robotic arm 110 may, for example, repeat this process for each of the containers 104 stored within the plate 102.

[0022] To determine whether a given container should be rejected, the controller may determine some attributes of the container and / or the sample. The controller may determine the attributes by analyzing an image of the container and / or the sample, including images corresponding to different orientations of the container 114, using a pattern recognition model. In some embodiments, the pattern recognition model may be a machine learning model (e.g., trained using a supervised learning method) or a fully human-designed model (e.g., a discovery model). The training of the machine learning model is described in more detail below.

[0023] As used herein, "attribute" can broadly mean any distinguishable quality or characteristic of a sample or container. For example, and without limitation, attributes of a fluid sample can be the presence of one or more objects (e.g., particles or bubbles), the presence of one or more specific types of objects (e.g., a specific type of particle or bubble, e.g., particles made of a specific material, or particles or bubbles having a specific size or within a specific size range, etc.), the number of objects or specific types of objects, the size of the objects or specific types of objects, the presence or size of dirt, etc. As another example, attributes of a container can be the presence of one or more chips or cracks, the presence of one or more specific types of chips or cracks (e.g., straight vs. serrated cracks, or chips having at least a specific depth, etc.), the presence of one or more deformations (e.g., warping) in the container, the presence of deformations in specific parts of the container (e.g., plunger, flange, etc.), etc. Thus, the pattern recognition model can operate by identifying and / or classifying objects, dirt, chips, cracks, deformations, etc. As a mere example, in one embodiment, the model may determine that all of the specific pixels in a digital image captured by the imager of imaging system 106 correspond to a single object, and then classify the identified object as a bubble, a particle, a specific type of particle or bubble, etc. As used herein, the term "particle" refers to an object other than a gas-containing bubble, such as, for example, a proteinaceous particulate or a glass flake.

[0024] In some embodiments, the controller may be configured to determine only one attribute or several different attributes for a given container or sample. In some embodiments, for example, the pattern recognition model of the controller is trained or designed to determine a first set of attributes (e.g., to identify the presence of specific objects and classify these objects), and then the controller performs further processing to determine a second set of attributes (e.g., to determine the number of identified and classified objects, to measure the size of the identified objects, to generate a histogram of particle sizes, etc.).

[0025] Figure 2, which provides an enlarged view of the plunger 150 of a syringe filled with liquid, shows an example of an attribute of a fluid sample. For example, the syringe may be the container 114 of FIG. 1. As seen in FIG. 2, the side wall 152 surrounds the syringe contents (i.e., the fluid sample), and the surface 154 of the piston / dome of the plunger 150 is covered by several objects 160 including bubbles 160A-160E and particles 160F. The bubbles 160A-160E can be, for example, the result of agitation and / or chemical reactions. The particles 160F may be contaminants introduced during the manufacture or processing of the fluid sample (or during the manufacture of the plunger 150, etc.) and may be harmless or harmful. Other examples of possible defects (not shown in FIG. 2) may include dirt and / or other attributes.

[0026] Referring again to FIG. 1, the controller may reject a given container if the container and / or the sample within the container do not meet one or more criteria. This criterion is at least partly based on the determined attributes. The criterion may be relatively simple (e.g., if particles or other non-bubble objects are present in the sample, the fluid sample is rejected), or may be more complex (e.g., if the "score" of the sample exceeds a threshold, the fluid sample is rejected. In this case, the controller calculates the score based on the number, type, and size of the particles in the sample and the number, type, and size of the bubbles in the sample). In embodiments where the controller can inspect different types of containers and / or samples (and / or different fields of view / viewpoints, etc.), the controller may store different sets of criteria for different types of containers and / or samples (and / or different fields of view / viewpoints, etc.).

[0027] Here, an exemplary embodiment of the controller will be described with reference to FIG. 3. As seen in FIG. 3, the control system 200 may include a controller 206 (e.g., the controller described above). The controller 206 is configured to communicate with and control various components of the robot inspection platform 100 of FIG. 1, which includes the robot arm 110 and the imaging system 106. In other embodiments, the control system 200 may be used to control a robot inspection platform different from the platform 100 of FIG. 1.

[0028] In one embodiment, the control system 200 is configured to facilitate fully autonomous or semi-autonomous operation of the robot inspection platform 100. In particular, the control system 200 may support the automated operation and analysis of containers, such as container 104 (e.g., in the manner described above). The controller 206 may be implemented, for example, as any suitable combination of hardware and / or software coupled to or otherwise in communication with the robot arm 110 and the imaging system 106. For example, the controller 206 may be implemented as a device attached to or integrated within a stage to which the robot arm 110 and / or the imaging system 106 are attached. Alternatively, the controller 206 may be located remotely from the robot inspection platform 100.

[0029] In any case, the controller 206 may be coupled to the robotic arm 110 and / or the imaging system 106 via a wired link, a wireless link, or any suitable combination thereof. Thus, in FIG. 3, the link 210 and / or the link 212 may each represent one or more wired and / or wireless links for facilitating communication between the controller 206 and one or both of the robotic arm 110 and the imaging system 106. Although FIG. 3 shows two separate links 210 and 212, it is understood that the controller 206 may communicate with the robotic arm 110 and the imaging system 106 via any suitable number of links, such as a single common link or bus, for example.

[0030] To facilitate communication with and control of the robotic arm 110 and the imaging system 106, the controller 206 may include a processing unit 220, a communication unit 222, and a memory unit 224. The processing unit 220 may be implemented as any suitable type and / or number of processors, such as, for example, the host processor of the controller 206. To provide further examples, the processing unit 220 may be implemented as an application specific integrated circuit (ASIC), an embedded processor, a central processing unit (CPU), etc., associated with the controller 206. Although not shown in FIG. 3, the processing unit 220 may be coupled to (and / or otherwise configured to communicate with, control, operate with, and / or affect the operation of) the communication unit 222 and / or the memory unit 224 via one or more wired and / or wireless interconnects, such as any suitable number of data and / or address buses, for example.

[0031] For example, the processing unit 220 may be configured to search, process, and / or analyze data stored in the memory unit 224 (e.g., images captured by one or more imagers of the imaging system 106), may be configured to store such data in the memory unit 224, may be configured to replace data stored in the memory unit 224, may be configured to control various functions related to the robotic arm 110 (e.g., any of the functions of the robotic arms described herein) and / or the imaging system 106 (e.g., any of the imaging functions described herein), and so on.

[0032] The communication unit 222 may be configured to support any suitable number and / or type of communication protocols for facilitating communication between the controller 206 and one or both of the robotic arm 110 and the imaging system 106. The communication unit 222 may be configured to facilitate the exchange of any suitable type of information (e.g., via the link 210 and / or the link 212) between the controller 206 and one or both of the robotic arm 110 and the imaging system 106, and may be implemented as any suitable combination of hardware and / or software for facilitating such functions. For example, the communication unit 222 may implement any number of wired and / or wireless transceivers, modems, ports, input / output interfaces, connectors, antennas, etc. Although not shown in FIG. 3, the robotic arm 110 and / or the imaging system 106 may include a similar type of communication unit to enable communication via the link 210 and / or the link 212.

[0033] According to various embodiments, the memory unit 224 may be one or more computer-readable non-transitory storage devices including any suitable volatile memory (e.g., random access memory (RAM)), non-volatile memory (e.g., battery-backed RAM, FLASH, etc.), or combinations thereof. The memory unit 224 may be configured to store instructions executable by the processing unit 220. These instructions may include machine-readable instructions that, when executed by the processing unit 220, cause the processing unit 220 to perform various acts described herein. The various functions of the controller 206 are described herein in terms of execution by the processing unit 220 of instructions stored in the memory unit 224, but it will be understood that equivalently, the functions may instead be implemented using only hardware components (e.g., a hardware processor). The memory unit 224 may also be configured to store any other suitable data used with the robotic inspection platform 100, such as images captured by the imager of the imaging system 106, data indicating identified samples and / or attributes of the containers, etc.

[0034] The control module 230 is a series of instructions within the memory unit 224 that, when executed by the processing unit 220, cause the processing unit 220 to perform various acts according to suitable embodiments described herein. In one embodiment, the control module 230, when executed by the processing unit 220, includes instructions that cause the processing unit 220 to send to the robotic arm 110 one or more commands for controlling the operation (e.g., movement) of the robotic arm 110 as described above in connection with FIG. 1. Further, the control module 230 may include instructions that, when executed by the processing unit 220, cause the processing unit 220 to send to the imaging system 106 one or more commands for controlling the operation of the imaging system 106 as described above in connection with FIG. 1.

[0035] For example, the processing unit 220 may transmit commands to the robotic arm 110 via the communication unit 222. These commands cause the robotic arm 110 to sequentially remove the containers 104 from the plate 102, place each container 104 within the field of view of one or more imagers of the imaging system 106, and manipulate each container 104 to assume a plurality of orientations while within the field of view of the imager. Alternatively, the processing unit 220 may transmit commands to the imaging system 106 via the communication unit 222. These commands cause the imager (or imagers) to be moved relative to a stationary container (e.g., rotated around the container while facing the container such that the container remains within the field of view). In either case, over the same time frame, the processing unit 220 may transmit commands to the imaging system 106 via the communication unit 222. These commands cause the imager to capture images of each container 104 while in each of the relative orientations described above, and cause these images to be provided to and stored in the controller 206 and the memory unit 224. In some embodiments, the processing unit 220 also transmits commands to the imaging system 106 via the communication unit 222. These commands cause the light source of the imaging system 106 to be turned on and off in a timely manner (e.g., such that each container 104 is illuminated when an image is captured), and / or cause other parameters of such a light source (e.g., intensity level, wavelength, etc.) to be set.

[0036] When executed by the processing unit 220, the nonconforming product identification module 232 is a series of instructions within the memory unit 224 that cause the processing unit 220 to perform various acts in accordance with the appropriate embodiments described herein. In one embodiment, when executed by the processing unit 220, the nonconforming product identification module 232 includes instructions that cause the processing unit 220 to process the container / sample images received from the imaging system 106 using the pattern recognition module 240.

[0037] The pattern recognition module 240 may process images (e.g., one per container, one per orientation, or multiple per orientation) using any suitable pattern recognition model and determine one or more attributes of the containers and / or samples corresponding to these images. In some embodiments, the pattern recognition module 240 implements a machine learning model trained using supervised learning as further described below. Generally, the pattern recognition model may use any suitable approach such as parametric or non-parametric classification algorithms, neural networks, support vector machines, etc. The pattern recognition module 240 is trained or designed to identify specific attributes such as, for example, any of the attributes described herein (e.g., an object, a particular type of object, a structural defect, etc.).

[0038] In some embodiments, the pattern recognition module 240 implements a trained machine learning model that receives an input (e.g., one or more images of a container), passes the input through the trained model once, and generates an output (e.g., one or more attribute classifications regarding the container and / or the sample within the container) in a process called performing "inference". Any suitable techniques or advancements regarding hardware or other processing units (e.g., FPGAs, ASICs, graphics processing units (GPUs), etc.) can be utilized to perform such inferences very quickly, resulting in a very fast and efficient inspection of the container and / or the sample.

[0039] In some embodiments, the pattern recognition module 240 outputs not only a classification / label / category indicating a particular container and / or sample attribute, but also a confidence score reflecting how likely the classification is correct. For example, the pattern recognition module 240 may output a value [particle, 0.67222] indicating that the probability of the presence of a particular particle in the sample is 67.222%, a value [particle A, 0.853] indicating that the probability of the sample containing a particular particle of type A is 85.3%, or a value [3, particle A, 0.510] indicating that the probability of the sample containing exactly three particles of type A is 51%, etc.

[0040] In some embodiments, the nonconforming product identification module 232 is configured to determine additional attributes not output by the pattern recognition module 240. For example, the nonconforming product identification module 232 may determine / measure the size of an object classified by the pattern recognition module 240 and / or may determine / count the number of objects within a particular class, etc. Whether or not the nonconforming product identification module 232 determines any additional attributes, the nonconforming product identification module 232 may use the determined attributes (and, if possible, the corresponding confidence scores as well) to determine whether the container should be marked as nonconforming, i.e., whether the container and / or sample fails to meet one or more criteria.

[0041] For example, the nonconforming product identification module 232 may mark as nonconforming a container holding a sample in which the pattern recognition module 240 has identified a threshold number of particles and / or bubbles, a threshold number of a particular type of particle and / or bubble, or a threshold number of a particular type of bubble having at least a threshold size, etc. in the corresponding image. In some embodiments, the nonconforming product identification module 232 calculates a score that is a mathematical function of the attributes and confidence scores. If the score exceeds (or falls below) a predetermined threshold, the container is marked as nonconforming. After the nonconforming product identification module 232 has determined whether a given container should be marked as nonconforming, the processing unit 220 may send, via the communication unit 222, a command to the robotic arm 110 to place the container in the nonconforming product bin 120 (if marked as nonconforming) or return it to the plate 102 or another appropriate location (if not marked as nonconforming).

[0042] In some embodiments, as described above, the robotic inspection platform 100 may be used not only to analyze samples (and / or the containers themselves) using a trained machine learning model, but also to facilitate the training of such a model. In such embodiments, the processing unit 220 may perform a process in which an image of a training container (i.e., a container holding a training sample or an empty training container) is captured, presented to a human analyst for manual labeling, and the labeled images are used in a supervised learning process. In particular, the processing unit 220 may send commands to the robotic arm 110 via the communication unit 222, which commands the robotic arm 110 to sequentially retrieve training containers from a plate (e.g., similar to plate 102) or other suitable source area, place each training container within the field of view of the imager of the imaging system 106, and manipulate each training container to take multiple orientations while within the field of view of the imager. Over the same time frame, the processing unit 220 may also send commands to the imaging system 106 via the communication unit 222, which commands the imager of the imaging system 106 to capture images of each training container while the container (or the imager) is in several different orientations, cause these training images to be provided to the controller 206, and stored in the memory unit 224. In some embodiments, the processing unit 220 may also send commands to the imaging system 106 via the communication unit 222, which commands the imaging system 106 to turn the light source on and off in a timely manner (e.g., such that each training container is illuminated when a training image is captured), and / or set other parameters of such a light source (e.g., intensity level, wavelength, etc.).

[0043] Generally, the processing unit 220 may operate the robot inspection platform 100 in the same manner as during runtime operation. For example, the training sample container (or imager) may be operated to take the same orientation as will later be used during runtime operation, as described above. If the light source has computer-controllable parameters (e.g., intensity, wavelength, etc.), the training process may also include the processing unit 220 sending commands to cause the same light source parameters to be present during training as will later be used during runtime operation. Regarding any parameters or configurations of the manually adjustable light source, it is generally important that the parameters / configurations be the same or at least very similar during both training and runtime operation.

[0044] A human analyst may label the training images in various ways depending on the embodiment. For example, the processing unit 220 may store the training images in the memory unit 224 and generate a graphical user interface (GUI) that presents the images stored for a given training container. The processing unit 220 may also provide various interactive features to assist the analyst's labeling process. In one such embodiment, the analyst may drag or draw a rectangle or other polygon around an object or other attribute in the image and type (or select from a menu, etc.) the classification of the enclosed attribute. Referring to FIG. 2, for example, an analyst looking at an image of the syringe 150 shown may draw a rectangle around the particle 160F and type or select the type of particle that the analyst believes accurately depicts the particle 160F. As another example, the analyst may draw a rectangle around the bubble 160C and type or select "bubble" (or a particular type of bubble) as a label, etc.

[0045] FIG. 4 is a flow diagram showing an exemplary method 300 for performing an inspection of a container (i.e., the sample within the container and / or the container itself) using pattern recognition, according to one embodiment. Method 300 may be performed, in whole or in part, by, for example, controller 206, robotic arm 110, and / or imaging system 106. In some embodiments, when instructions stored in memory unit 224 are executed, all of method 300 is performed by processing unit 220 of controller 206.

[0046] In method 300, the robotic arm is caused to remove a container using the end effector of the robotic arm (block 302). The robotic arm may be similar to robotic arm 110 of FIGS. 1 and 3, for example, and the end effector may be similar to end effector 112 of FIG. 1. The container may be a syringe, vial, cartridge, or any other type of container that may or may not contain a sample (e.g., a fluid or lyophilized product), depending on the embodiment. Block 302 may be performed, for example, by generating one or more command signals and transmitting the signals to the robotic arm (e.g., a slave controller within the robotic arm).

[0047] The robotic arm is caused to manipulate the container using the end effector such that the container is sequentially positioned in a plurality of orientations while the container is within the field of view of an imager (e.g., the imager of imaging system 106 of FIG. 1) (block 304). In some embodiments, the container is simultaneously positioned within the field of view of one or more additional imagers (e.g., a total of three imagers spaced 120 degrees apart in a two-dimensional plane). Block 304 may be performed, for example, by generating one or more command signals and transmitting the signals to the robotic arm (e.g., a slave controller within the robotic arm).

[0048] Cause the imager to capture a plurality of images (block 306), each image being captured while the container is in each of the orientations. The orientations may include, for example, different rotational positions (e.g., every 20 degrees or every 90 degrees of a full circle, etc.) with respect to the longitudinal axis of the container. Block 306 may be implemented, for example, by generating one or more command signals and transmitting the signals to the imager (e.g., a slave controller within the imager). Block 306 may further include receiving the images captured by the imager and storing these images in a memory (e.g., memory unit 224 of FIG. 3).

[0049] In another embodiment, block 304 instead (or additionally) includes moving the imager (or imagers) relative to the container such that the relative orientation between the container and the imager changes without necessarily requiring any additional movement (e.g., rotation) of the container via the end effector of the robotic arm. For example, the container may remain stationary once initially placed in a predetermined position by the robotic arm, and the imager may move in a circular path around the container (e.g., in 20-degree increments of a full circle). In such an embodiment, block 306 may include causing the imager to capture one or more images at each position of the imager.

[0050] One or more attributes of the container and / or sample are determined (block 308) by analyzing the images captured in block 306 using a pattern recognition model. The attributes may be determined by analyzing the container and / or sample using a model trained to classify the attributes of the container and / or sample using, for example, a machine learning model (e.g., using a supervised learning method). The attributes may include the presence of a general object, the presence of a particular type of object, the presence of dirt or a certain type of dirt, the presence of cracks or deformations in the container, and / or other attributes.

[0051] Based on the attributes determined in block 308, it is determined whether the container and / or sample meets one or more criteria (block 310). The criteria may include, for example, at least one criterion according to the type of object (e.g., whether the sample contains particles, bubbles, a specific type of particle or bubble, etc.). The one or more criteria may also take into account a confidence level / score associated with one or more of the attributes.

[0052] Based on whether one or more criteria are met, the container is placed in an area reserved for non-conforming containers and / or samples (e.g., the non-conforming product box 120 in FIG. 1) or in an area reserved for containers and / or samples that did not fail (e.g., the plate 102 in FIG. 1) by the robotic arm (block 312). Block 312 may be implemented, for example, by generating one or more command signals and transmitting the signals to the robotic arm (e.g., a slave controller within the robotic arm).

[0053] In some embodiments, method 300 includes one or more additional blocks not shown in FIG. 4. In some embodiments where the machine learning model is used at block 306, for example, method 300 may include some additional blocks corresponding to the training of the machine learning model before block 302. For example, method 300 may include additional blocks that cause a robotic arm to sequentially remove a plurality of training containers using an end effector. For each of the removed training containers, the method includes causing the robotic arm to manipulate the container using the end effector such that the container is sequentially arranged in a plurality of orientations while the container is within the field of view of the imager, causing the imager to capture each training image (each of the training images is captured while the training container is in each one of the orientations), and presenting each image to a user via a display to facilitate manual labeling of attributes regarding the training container and / or samples within the training container. These additional blocks may include, for example, generating a command signal and transmitting the signal to an appropriate destination (e.g., a slave controller of the robotic arm and a slave controller of the imager).

[0054] FIG. 5 is a flowchart of an exemplary method 350 for performing an inspection of a series of containers, according to one embodiment. Method 350 may be performed, in whole or in part, by, for example, controller 206 (e.g., processing unit 220), robotic arm 110, and / or imaging system 106. As further described below, method 350 may be used in conjunction with method 300 of FIG. 4.

[0055] Method 350 may be initiated by capturing images of containers in different orientations (block 352) and processing the captured images to determine the attributes of the containers (and / or samples within the containers) and whether one or more criteria are met (block 354). The containers may be classified as non-conforming if one or more criteria are not met or as conforming if one or more criteria are met. For a given container, for example, block 352 may correspond to blocks 302-306 of method 300, and block 354 may correspond to blocks 308 and 310 of method 300.

[0056] Once a given container has been analyzed and classified as either non-conforming or conforming, method 350 may include determining whether all of the containers selected for analysis have already been analyzed (block 356). This determination may be made, for example, by tracking the total number of containers analyzed for a given type of plate having a known number of containers (e.g., 24 or 96) and determining whether this number is less than or equal to the total number of containers. Alternatively, the determination may be made, for example, by having the user place any suitable number of containers (which may be less than the total number of containers in the plate) for analysis, tracking the total number of containers analyzed, and determining whether this number is less than or equal to the number of test containers placed.

[0057] In any case, if there is a need to analyze a further container, method 350 continues by removing the next container (block 358). Once the next container is properly positioned, method 350 may repeat the process by capturing an image of the container and / or the sample within the container (block 352). Conversely, if the analyzed container is the last container to be analyzed, method 350 ends. Upon ending (or in parallel with the iteration of block 354), method 350 may include storing or outputting an analysis report for each container (e.g., to memory unit 224 as shown in FIG. 3). Once this report is complete, the user may view the report and / or initiate the process for a new plate.

[0058] Some of the figures described herein show exemplary block diagrams having one or more functional components. Such block diagrams are for illustrative purposes, and it will be understood that the devices described and illustrated may have more, fewer, or different components than those shown. Further, in various embodiments, the components (and the functions provided by each component) may be associated or integrated as part of any suitable component.

[0059] Embodiments of the present disclosure relate to a non-transitory computer-readable storage medium having computer code for performing various operations implemented by a computer. As used herein, the term "computer-readable storage medium" includes any medium that can store or encode a set of instructions or computer code for performing the operations, techniques, and methods described herein. The medium and the computer code may be specially designed and constructed for the embodiments of the present disclosure, or may be of the kind well-known and available to those of ordinary skill in the computer software arts. Examples of computer-readable storage media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs and holographic devices; magneto-optical media such as optical disks; and hardware devices specially configured to store and execute program code, such as ASICs, programmable logic devices ("PLDs"), and ROM and RAM devices, but are not limited thereto.

[0060] Examples of computer code include machine code such as generated by a compiler, and files containing higher-level code that is executed by a computer using an interpreter or compiler. For example, embodiments of the present disclosure may be implemented using Java, C++, or other object-oriented programming languages and development tools. Further examples of computer code include encrypted code and compressed code. Additionally, embodiments of the present disclosure may be downloaded as a computer program product, which may be transferred from a remote computer (e.g., a server computer) to a requesting computer (e.g., a client computer or a different server computer) via a transmission channel. Another embodiment of the present disclosure may be implemented in a hard-wired circuit instead of or in combination with machine-executable software instructions.

[0061] As used herein, the singular terms "a", "an", and "the" may include plural referents unless the context clearly dictates otherwise.

[0062] As used herein, relative terms such as "above", "below", "up", "left", "right", "down", "top", "bottom", "vertical", "horizontal", "side", "higher", "lower", "upper", "over", "under", "inner", "interior", "outer", "exterior", "front", "back", "upwardly", "downwardly", "vertically", "lateral", "laterally", etc. refer to the orientation of a series of components relative to each other. This orientation follows the drawings but is not required during manufacture or use.

[0063] As used herein, the terms "connect", "connected", and "connection" refer to operative coupling or linkage. The components being connected may be directly coupled to each other or indirectly coupled, for example, via another series of components.

[0064] As used herein, the terms "approximately", "substantially", "substantial", and "about" are used to describe and account for minor variations. These terms, when used with an event or circumstance, can mean that the event or circumstance occurs precisely, and that the event or circumstance occurs in close approximation. For example, when used with a numerical value, these terms can mean a variation range of up to ±10% of that numerical value, such as up to ±5%, ±4%, ±3%, ±2%, ±1%, ±0.5%, ±0.1%, or ±0.05% thereof. For example, two numerical values can be considered "substantially" the same if the difference between these values is up to ±10% of the average of these values, such as up to ±5%, ±4%, ±3%, ±2%, ±1%, ±0.5%, ±0.1%, or ±0.05% thereof.

[0065] Furthermore, amounts, ratios, and other numerical values may be presented herein in a range format. Such a range format is used for convenience and brevity and is to be interpreted flexibly as including the numerical values explicitly recited as the limits of the range, as well as every individual numerical value or sub-range subsumed within that range as if each numerical value and sub-range were explicitly recited.

[0066] Although the present disclosure has been described and illustrated with reference to specific embodiments thereof, such description and illustration are not intended to be limiting. Those skilled in the art should understand that various changes may be made and equivalents may be substituted without departing from the true spirit and scope of the present disclosure as defined by the appended claims. The figures may not be drawn to scale. Differences may exist between the artistic renditions in the present disclosure and the actual devices due to manufacturing processes and tolerances. Other embodiments of the present disclosure that are not specifically illustrated may exist. The specification and drawings should be regarded as illustrative rather than restrictive. Modifications may be made to adapt a particular situation, material, composition of matter, method, or process to the purposes, spirit, and scope of the present disclosure. All such modifications are intended to be included within the scope of the appended claims. Although the methods disclosed herein have been described with reference to specific operations performed in a particular order, it will be understood that these operations may be combined, re-divided, or rearranged to form equivalent methods without departing from the teachings of the present disclosure. Accordingly, the order and grouping of operations are not intended to limit the present disclosure unless explicitly stated herein.

Claims

1. a robotic arm including an end effector and a number of articulated segments; an imaging system including an imager; A controller, having the robotic arm retrieve a container using the end effector; causing the robotic arm to manipulate the container using the end effector such that the container is sequentially positioned in a plurality of orientations while the container is within a field of view of the imager; causing the imager to capture a plurality of images, each of the plurality of images being captured while the container is in a respective one of the plurality of orientations; determining one or more attributes of the container and / or a sample within the container by analyzing the plurality of images using a pattern recognition model; determining whether the container and / or the sample meets one or more criteria based on the one or more attributes of the container and / or the sample; Based on whether the container and / or sample meets the one or more criteria, either (i) cause the robotic arm to place the container in an area reserved for rejected containers and / or samples, or (ii) cause the robotic arm to place the container in an area reserved for non-rejected containers and / or samples. A controller configured to: ,Robotic inspection platform.

2. 2. The robotic inspection platform of claim 1, wherein the controller is configured to determine the one or more attributes of the container and / or the sample by analyzing the plurality of images using a machine learning model, the machine learning model being trained to classify attributes of the container and / or sample.

3. The controller includes at least causing the robot arm to sequentially retrieve a plurality of training containers using the end effector; For each of the plurality of removed training containers, (i) causing the robot arm to manipulate the training container using the end effector such that the training container is sequentially positioned in the plurality of orientations while the training container is within a field of view of the imager; (ii) causing the imager to capture a respective plurality of training images, each of the respective plurality of training images being captured while the training container is in a respective one of the plurality of orientations; and (iii) presenting each of the respective plurality of images to a user via a display to facilitate manual labeling of attributes related to the training container and / or samples within the training container.

3. The robotic inspection platform of claim 2, configured to facilitate training of the machine learning model by:

4. 4. The robotic inspection platform of claim 1, wherein the controller is configured to distinguish between different object types by analyzing the plurality of images using the pattern recognition model.

5. The robotic inspection platform of claim 4 , wherein the different object types include bubbles and particles.

6. The robotic inspection platform of claim 4 or 5, wherein the different object types include one or both of: (i) a specific type of bubbles; and (ii) a specific type of particles.

7. The robotic inspection platform of any one of claims 4 to 6, wherein the one or more criterion includes at least one criterion depending on a type of object.

8. The robotic inspection platform of claim 1 , wherein the multiple orientations include multiple rotations about the longitudinal axis of the container.

9. The robotic inspection platform of any one of claims 1 to 8, wherein the area reserved for rejected containers and / or samples is a box.

10. The robotic inspection platform of any one of claims 1 to 9, wherein the controller is configured to cause the robotic arm to remove the container from a plate.

11. The robotic inspection platform of claim 10 , wherein the area reserved for non-rejected containers and / or samples is an area within the plate.

12. 1. A method for performing inspection of a container using pattern recognition, comprising: causing a robotic arm to retrieve a container using an end effector of the robotic arm; causing the robotic arm to manipulate the container using the end effector such that the container is sequentially positioned in a plurality of orientations while the container is within a field of view of an imager; causing the imager to capture a plurality of images, each of the plurality of images being captured while the container is in a respective one of the plurality of orientations; determining one or more attributes of the container and / or a sample within the container by analyzing the plurality of images using a pattern recognition model; determining whether the container and / or the sample meets one or more criteria based on the one or more attributes of the container and / or the sample; Based on whether the container and / or sample meets the one or more criteria, either (i) causing the robotic arm to place the container in an area reserved for rejected containers and / or samples, or (ii) causing the robotic arm to place the container in an area reserved for non-rejected containers and / or samples; A method comprising:

13. Determining the one or more attributes of the fluid sample by analyzing the plurality of images using a pattern recognition model includes: determining the one or more attributes of the container and / or the sample by analyzing the plurality of images using a machine learning model, the machine learning model being trained to classify attributes of the container and / or the sample. The method of claim 12, comprising:

14. at least, causing the robot arm to sequentially retrieve a plurality of training containers using the end effector; For each of the plurality of removed training containers, (i) causing the robot arm to manipulate the training container using the end effector such that the training container is sequentially positioned in the plurality of orientations while the training container is within a field of view of the imager; (ii) causing the imager to capture a respective plurality of training images, each of the respective plurality of training images being captured while the training container is in a respective one of the plurality of orientations; and (iii) presenting each of the respective plurality of images to a user via a display to facilitate manual labeling of attributes related to the training container and / or samples within the training container.

14. The method of claim 13, further comprising facilitating training of the machine learning model by:

15. 15. A method according to any one of claims 12 to 14, wherein determining one or more attributes of the container and / or the sample by analysing the plurality of images using the pattern recognition model comprises distinguishing between different object types by analysing the plurality of images using the pattern recognition model.

16. The method of claim 15 , wherein the one or more criteria include at least one criterion according to a type of object.

17. The method of any one of claims 12 to 16, wherein the multiple orientations include multiple rotations about the longitudinal axis of the container.

18. 1. A method for performing inspection of a container using pattern recognition, comprising: causing a robotic arm to retrieve a container using an end effector of the robotic arm; sequentially positioning an imager in a plurality of orientations while the container is within a field of view of the imager; causing the imager to capture a plurality of images of the container, each of the plurality of images being captured while the imager is in a respective one of the plurality of orientations; determining one or more attributes of the container and / or the sample by analyzing the plurality of images using a pattern recognition model; determining whether the container and / or the sample meets one or more criteria based on the one or more attributes of the container and / or the sample; Based on whether the container and / or sample meets the one or more criteria, either (i) causing the robotic arm to place the container in an area reserved for rejected containers and / or samples, or (ii) causing the robotic arm to place the container in an area reserved for non-rejected containers and / or samples; A method comprising:

19. Determining the one or more attributes of the container and / or the sample by analyzing the plurality of images using a pattern recognition model includes: determining the one or more attributes of the container and / or the sample by analyzing the plurality of images using a machine learning model, the machine learning model being trained to classify attributes of the container and / or the sample.

20. The method of claim 18, comprising:

20. at least, causing the robot arm to sequentially retrieve a plurality of training containers using the end effector; For each of the removed training containers, (i) sequentially positioning the imager in the multiple orientations while the training container is within the field of view of the imager; (ii) causing the imager to capture a respective plurality of training images, each of the respective plurality of training images being captured while the imager is in a respective one of the multiple orientations; and (iii) presenting each of the respective plurality of images to a user via a display to facilitate manual labeling of attributes related to the training container and / or samples within the training container.

20. The method of claim 19, further comprising facilitating training of the machine learning model by:

Images