Robotic system for performing pattern recognition-based inspection of pharmaceutical containers

The robotic inspection platform with pattern recognition models effectively distinguishes between particles and bubbles in pharmaceutical containers, enhancing inspection accuracy and reducing risks and costs.

JP7954234B2Active Publication Date: 2026-11-19AMGEN INC
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
JP2025049364
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2018-07-31
Filing Date
2025-03-25
Publication Date
2026-11-19
Estimated Expiration
2039-06-12

AI Technical Summary

Technical Problem

Conventional image processing methods for inspecting pharmaceutical containers struggle with high accuracy when distinguishing between defects like particles and non-defects, such as bubbles, leading to potential safety risks or unnecessary disposal costs.

Method used

A robotic inspection platform using pattern recognition models, trained with supervised learning, to accurately differentiate between particles and bubbles in pharmaceutical containers, employing a robotic arm for manipulation and an imaging system for detailed image capture.

Benefits of technology

Ensures reliable detection of defects with high accuracy, reducing safety risks and costs by automating the inspection process while minimizing false rejections and approvals.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007954234000001
    Figure 0007954234000001
  • Figure 0007954234000002
    Figure 0007954234000002
  • Figure 0007954234000003
    Figure 0007954234000003
Patent Text Reader

Abstract

To address challenging inspection tasks by performing the inspection of pharmaceutical containers and / or the inspection of samples within such containers.SOLUTION: A robotic inspection platform comprises a robotic arm, an imager, and a controller. The controller causes the robotic arm to use its end effector to retrieve a container, and to manipulate the container such that the container is sequentially placed in a plurality of orientations while in view of the imager. The controller also causes the imager to capture images, with each of the images being captured while the container is in a respective one of the orientations. The controller also determines one or more attributes of the container, and / or a sample within the container, by analyzing the images using a pattern recognition model and, based on the attributes, determines whether the container and / or sample satisfies one or more criteria. If the container and / or sample fails to satisfy the criteria, the controller causes the robotic arm to place the container in an area reserved for rejected containers and / or samples.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The present application generally relates to inspection of pharmaceutical containers such as syringes or vials, and / or inspection of samples within such containers. BACKGROUND ART

[0002] In certain contexts, such as quality control procedures for manufactured medicaments, fluids or other (e.g., lyophilized) samples need to be examined for the presence of various attributes such as particles, bubbles, or contaminants. Undesirable particles or other attributes may originate from many different causes, such as the environment, improper handling or storage, or by-products / residues from formation, packaging, or filling. Depending on the circumstances, certain types of attributes (e.g., bubbles) may be acceptable, while other attributes (e.g., particles, specific types of particles, particles exceeding a specific size and / or number, etc.) may be unacceptable. If unacceptable attributes are detected in a fluid sample, the fluid sample is rejected.

[0003] Image processing methods conventionally used for defect detection in large commercial inspection machines often struggle to achieve high accuracy when faced with challenging inspection tasks. This is especially true for inspections that require evaluating samples that have very similar appearances while simultaneously possessing both defective and non-defective attributes. For example, inspecting a liquid-filled syringe may involve distinguishing between particles or contaminants (defects) on the piston dome and harmless bubbles (non-defects) on the piston dome. In automated processes, this can make it difficult to ensure that defective samples are rejected at an acceptable rate (e.g., 100% probability or at least 99.99999% probability), and also difficult to ensure that qualified samples are approved at an acceptable rate (i.e., not mistakenly rejected). The former situation can lead to potentially dangerous conditions, while the latter can result in high costs due to the unnecessary disposal of accepted products. The low accuracy of automated systems can sometimes be compensated for by other means (e.g., repeated inspections, manual inspections), but such methods are generally expensive in terms of both time and cost. Similar drawbacks exist in relation to other quality procedures, such as inspecting for defects in the containers themselves. [Overview of the Initiative] [Means for solving the problem]

[0004] The embodiments described herein relate to systems that improve upon conventional inspection methods. In particular, robotic inspection platforms can use pattern recognition (e.g., deep learning) models / methods to inspect pharmaceutical containers (e.g., syringes, vials, cartridges, etc.) and / or samples (e.g., fluids, lyophilized products, etc.) within such containers, and can address even very difficult inspection tasks. The pattern recognition model may be capable of reliably distinguishing, for example, between particles and bubbles in a fluid sample, or between specific types of particles and specific types of bubbles, etc. The robotic inspection platform may also be configured to facilitate the initial (pre-production) development or training of the pattern recognition model. In some embodiments and / or situations, the platform may be used to perform primary (e.g., single or initial) inspection of containers and / or samples. Alternatively, the platform may be used to re-inspect containers and / or samples that have already been identified as defective or likely to be defective by another automated inspection system (e.g., a conventional automated inspection system with lower reliability / accuracy).

[0005] In embodiments where the platform implements a machine learning model, the model may be trained using a large dataset of training images and a supervised learning method. The platform may be used under controlled conditions (e.g., accurate and reproducible container orientation and lighting conditions) to capture training images, after which a human analyst may manually label the attributes of the training images. For example, the analyst may label / classify objects in the training images as specific object types. To capture training images, the platform's robotic arm may sequentially pick up each of several containers, present the containers in front of the imager, and manipulate (e.g., rotate) the containers while images from different angles / orientations are acquired. In the training process, containers with intentionally introduced defects (and / or holding samples with intentionally introduced defects) may be used to ensure that a sufficiently wide range of defects and / or types of defects are encountered when a human analyst performs a classification task.

[0006] Once the model is trained and incorporated, the platform can be switched to “production,” “runtime,” “action,” or “processing” mode. In this mode, the robotic arm picks up a new container (which may contain a sample) and manipulates it in front of the imager, as it did during the training process. Images for each container are stored in memory and may be processed by a computer using the trained pattern recognition model. The computer may also process the model’s output by applying one or more criteria to determine whether the container is “pass” or “fail.” If the container is deemed “fail” (due to a defective sample in the container and / or the container itself being defective), the robotic arm may place the container in a box or other area specifically reserved for that purpose. Depending on the embodiment and / or circumstances, the robotic inspection platform may be configured to inspect only containers, only product samples (e.g., the fluid or freeze-dried product in the container), or both containers and samples. In one embodiment, a single robotic inspection platform can perform any of these functions, provided that a suitable machine learning model is introduced.

[0007] Those skilled in the art will understand that the drawings included herein are for illustrative purposes only and do not limit the disclosure. The drawings are not necessarily to scale and instead focus on illustrating the principles of the disclosure. In some cases, various aspects of the embodiments described may be exaggerated or enlarged to facilitate understanding of the embodiments described. In the drawings, similar reference numerals in different drawings refer to components that are generally functionally and / or structurally similar. [Brief explanation of the drawing]

[0008] [Figure 1] An exemplary robotic inspection platform according to one embodiment is shown. [Figure 2]Figure 1 shows a diagram of a plunger with a filled syringe sample as an exemplary syringe that can be inspected using the robotic inspection platform. [Figure 3] This is a block diagram of an exemplary control system that may be associated with the robot inspection platform of Figure 1, according to one embodiment. [Figure 4] This is a flowchart illustrating an exemplary method for performing inspection of containers and / or samples using pattern recognition, according to one embodiment. [Figure 5] This is a flowchart illustrating an exemplary method for performing a series of tests on a set of containers and / or samples contained within those containers, which may be used in conjunction with the method shown in Figure 4. [Modes for carrying out the invention]

[0009] The various concepts introduced above and described in more detail below can be implemented in any of many ways, and the concepts described are not limited to any particular implementation. Examples of embodiments are provided for illustrative purposes only.

[0010] Figure 1 shows a robotic inspection platform 100 according to one embodiment of the present disclosure. The robotic inspection platform 100 includes a plate 102 that houses several containers 104. The containers 104 may be fully or partially transparent or translucent pharmaceutical containers such as syringes, vials, cartridges, etc., each of which may contain a sample (e.g., a fluid or lyophilized medical product). Alternatively, in some embodiments and / or situations (e.g., when the robotic inspection platform 100 analyzes only the containers themselves and not the sample inside), the containers 104 may be opaque. Each container 104 may be configured to accept any appropriate amount of sample, depending on the size and configuration of the plate 102. For example, each container 104 may hold tens of nanoliters of fluid, or several milliliters of fluid, etc. The plate 102 may include several cavities or wells (e.g., 80 wells, 96 wells, etc.) sized to securely but removably hold the containers 104.

[0011] The robotic inspection platform 100 also includes an imaging system 106. The imaging system 106 is generally configured to illuminate a container, such as a vessel 104, and to capture an image of the illuminated vessel. For this purpose, the imaging system 106 may include a light source, an imager, and, if possible, one or more mirrors and / or other optical elements to redirect the light from the light source and / or the light scattered by the sample (and / or the container itself) in a suitable manner. The light source may be a light-emitting diode (LED) light or any other suitable type of light source, and may provide light of any suitable wavelength (or range of wavelengths), and may include lights in one or more positions. The light source may have only fixed or manually adjustable parameters, or may have one or more computer-controlled parameters (e.g., intensity, direction, etc.). Figure 1 shows only a single imager and one mirror, but instead, the imaging system 106 may include multiple imagers, multiple mirrors (or no mirrors), etc., arranged in different positions. The imager may include any suitable combination of hardware and / or software, such as an image sensor, optical stabilizer, image buffer, frame buffer, frame grabber, charge-coupled device (CCD), complementary metal-oxide-semiconductor (CMOS) device, etc. In some embodiments, the imager may include imaging devices operating at invisible wavelengths, such as one or more infrared camera devices. Furthermore, in some embodiments, the light source may include several lights having different configurations, orientations, parameters, etc., and the light source may be configured so that different lights can strobe. In this way, different images corresponding to different lights can be obtained. Different strobe lights may be more suitable, for example, for exposing different attributes of a container and / or sample.

[0012] The robot inspection platform 100 also includes a robotic arm 110. The robotic arm 110 has multiple articulated segments (e.g., seven segments capable of six degrees of freedom of motion) and an end effector 112. The joints connect the various segments, and the segments can be articulated using controllable stepping motors or other suitable means. The robotic arm 110 may incorporate motion sensors and / or other sensors to provide control feedback, ensuring that the robotic arm 110 moves in an intended state within a certain tolerance threshold.

[0013] Figure 1 shows an end effector 112 holding a container (e.g., a syringe) 114 that has been removed by a robot arm 110 from a container 104 in a plate 102. The end effector 112 has hardness suitable for firmly securing (e.g., gripping) the container, but also contains a material suitable for avoiding breakage of fragile (e.g., glass) containers. The end effector 112 may operate by compressing the container 114, or by inserting an element into the container 114 and expanding it outward. The end effector 112 may incorporate one or more pressure sensors to provide control feedback, ensuring that the end effector 112 provides an intended amount of force within a certain tolerance threshold.

[0014] The robotic arm 110 may include a motor or other suitable means for rotational / motion and / or translational / motion of the object held by the end effector 112. For example, the robotic arm 110 may be able to adjust the roll angle, pitch angle, and yaw angle of the container 114 by rotating and / or tilting the end effector 112 (for roll and pitch) and / or moving an articulated arm segment (for yaw). In some embodiments, the robotic arm 110 is a commercially available device such as the UR3 robotic arm offered by Universal Robots (UR®).

[0015] The robotic inspection platform 100 also includes a rejection box 120 for discarding defective containers and / or samples. In the embodiment shown in Figure 1, the rejection box 120 is a simple container having four side walls, a bottom / floor, and an open top. In other embodiments, the rejection box 120 may be constructed differently (for example, the container may include a top with an open slot, or a plate with wells similar to plate 102, etc.).

[0016] Although not shown in Figure 1, the controller is communicatively coupled to at least a portion of the imaging system 106 and the robot arm 110. The controller may be a general-purpose computing device specifically programmed according to the embodiments described herein, or it may be a dedicated controller device (e.g., a dedicated processor integrated within the imaging system 106 and / or the robot arm 110). One embodiment of the controller will be described in more detail below with reference to Figure 3.

[0017] The controller coordinates the operation of the imaging system 106 and the robotic arm 110 during runtime operation. As used herein, the term “runtime” operation refers to the operation of the production robot inspection platform (e.g., platform 100) for applying quality control procedures after the manufacture and packaging of pharmaceutical containers and / or samples, and before sale or other distribution. In some embodiments, as will be further detailed below, the controller also coordinates the operation of the imaging system 106 and the robotic arm 110 during training of a pattern recognition model prior to runtime operation.

[0018] During runtime operation, the controller generates command signals and transmits them to the imaging system 106 and / or the robot arm 110. The transmission of command / control signals is described further below with reference to Figure 3.

[0019] Herein, the control / operation of a robot inspection platform 100 according to one embodiment is described. First, the controller causes the robot arm 110 to remove the container 114 from the plate 102 by fixing the container 114 using the end effector 112 (e.g., gripping, suction, etc.) and move the container 114 to a position where it is clearly "visible" to the imager of the imaging system 106 (e.g., as shown in Figure 1). The light source of the imaging system 106 illuminates the container 114 and (if any) the sample inside the container 114. The various components of the imaging system 106 (e.g., imager, mirror, light source) are configured such that light from the light source passes through the container 114, is reflected, refracted or blocked by objects inside the container 114 (e.g., particles, bubbles, etc.), and enters the imager. Additionally or alternatively (e.g., if the container 114 does not contain a sample), light may be reflected, refracted or blocked by various parts of the container 114. For example, the light source may remain "on" for an extended period of time, or the controller may turn the light source on at appropriate intervals.

[0020] Subsequently, the controller causes the imager to capture at least one image of the container 114, and, if possible, the sample within the container 114. However, in some embodiments and / or applications where the container 114 holds the sample, the controller first causes the robotic arm 110 to agitate the container 114 (e.g., shake, invert, etc.) immediately before imaging. After the first image is captured, the controller causes the robotic arm 110 to operate the end effector 112 so that the container 114 is positioned in a series of different orientations. For example, the container 114 may be rotated in 20-degree increments about its longitudinal axis (e.g., providing a total of 18 different orientations) and / or tilted at various angles relative to the vertical position shown in Figure 1. This orientation may precisely correspond to the orientation used to obtain training images manually labeled by a human analyst, as further described below. In each orientation, the controller may cause the imager to capture one or more images of the container 114 and the fluid sample within the container 114. That is, the controller may synchronize the imaging system 106 and the robot arm 110 using command signals as needed. In some embodiments, different lights from the light source of the imaging system 106 (e.g., lights having different configurations, orientations, characteristics, etc.) strobe over each orientation of the container 114, and at least one image of the container 114 is captured for each light before the container 114 is moved to the next orientation. Alternatively, the robot arm 110 may move the container 114 over all orientations for a first light, and then move the container 114 over all orientations for a second light, and so on.

[0021] Thereafter, the controller may process the captured image to determine whether the container 114 (e.g., the container itself or the sample within the container 114) should be rejected. If rejected, the robotic arm 110 discards the container 114 by placing the container 114 into a rejected product bin 120. If not rejected, the robotic arm 110 may return the container 114 to the well of the plate 102 from which the container 114 was retrieved, or place the container 114 at another suitable position. The robotic arm 110 may, for example, repeat this process for each of the containers 104 stored in the plate 102.

[0022] To determine whether a given container should be rejected, the controller may determine a number of attributes of the container and / or the sample. The controller may determine the attributes by analyzing images of the container and / or the sample, including images corresponding to different orientations of the container 114, using a pattern recognition model. In some embodiments, the pattern recognition model may be a machine learning model (e.g., trained using a supervised learning method), or may be a fully human-designed model (e.g., a heuristic model). Training of the machine learning model is described in further detail below.

[0023] As used herein, “attribute” can broadly mean any verifiable quality or characteristic of a sample or container. For example, and without limitation, attributes of a fluid sample may include the presence of one or more objects (e.g., particles or bubbles), the presence of one or more objects of a particular type (e.g., a particular type of particle or bubble, e.g., particles made of a particular material, or particles or bubbles having or being within a particular size range), the number of objects or objects of a particular type, the size of objects or objects of a particular type, the presence or size of contamination, etc. Another example is that attributes of a container may include the presence of one or more chips or cracks, the presence of one or more chips or cracks of a particular type (e.g., a straight-to-sawtooth crack, or a chip having at least a certain depth, etc.), the presence of one or more deformations in the container (e.g., warping), the presence of deformation in a particular part of the container (e.g., a plunger, flange, etc.), etc. Thus, a pattern recognition model may operate by identifying and / or classifying objects, contamination, chips, cracks, deformations, etc. As merely one example, in one embodiment, the model may determine that all of a particular pixel in the digital image captured by the imager of the imaging system 106 corresponds to a single object, and then classify the identified object as a bubble, particle, a specific type of particle or bubble, etc. As used herein, the term "particle" refers to objects other than gaseous bubbles, such as proteinaceous particulate matter or glass flakes.

[0024] In some embodiments, the controller may be configured to determine only one attribute or several different attributes for a given container or sample. In some embodiments, for example, the controller's pattern recognition model is trained or designed to determine a first set of attributes (e.g., to identify the presence of specific objects and classify these objects), and the controller then performs further processing to determine a second set of attributes (e.g., to determine the number of identified and classified objects, to measure the size of identified objects, to generate a particle size histogram, etc.).

[0025] An example of attributes of a fluid sample is shown in FIG. 2, which provides an enlarged view of the plunger 150 of a syringe filled with liquid. For example, the syringe may be the container 114 of FIG. 1. As can be seen in FIG. 2, the sidewall 152 encloses the syringe contents (i.e., the fluid sample), and the piston / dome surface 154 of the plunger 150 is covered by a number of objects 160 including bubbles 160A to 160E and particles 160F. The bubbles 160A to 160E may be, for example, the result of agitation and / or a chemical reaction. The particles 160F may be contaminants introduced during the manufacture or processing of the fluid sample (or the manufacture of the plunger 150, etc.), and may be harmless or harmful. Other examples of potential defects (not shown in FIG. 2) may include smudges and / or other attributes.

[0026] Referring again to FIG. 1 herein, the controller may reject a given container if the container and / or the sample within the container do not meet one or more criteria. These criteria are based, at least in part, on the determined attributes. The criteria may be relatively simple (e.g., the fluid sample is rejected if particles or other non-bubble objects are present in the sample), or may be more complex (e.g., the fluid sample is rejected if the "score" of the sample exceeds a threshold, in which case the controller calculates the score based on the number, type, and size of particles in the sample and the number, type, and size of bubbles in the sample). In embodiments where the controller is capable of inspecting different types of containers and / or samples (and / or different fields of view / viewpoints, etc.), the controller may store different sets of criteria for different container and / or sample types (and / or different fields of view / viewpoints, etc.).

[0027] Here, an exemplary embodiment of the controller will be described with reference to Figure 3. As seen in Figure 3, the control system 200 may include a controller 206 (for example, the controller described above). The controller 206 is configured to communicate with and control various components of the robot inspection platform 100 of Figure 1, including the robot arm 110 and the imaging system 106. In other embodiments, the control system 200 may be used to control a robot inspection platform different from the platform 100 of Figure 1.

[0028] In one embodiment, the control system 200 is configured to facilitate fully autonomous or semi-autonomous operation of the robot inspection platform 100. In particular, the control system 200 may support the automatic operation and analysis of containers such as container 104 (for example, by the method described above). The controller 206 may be implemented as any suitable combination of hardware and / or software coupled to, for example, the robot arm 110 and the imaging system 106, or otherwise communicating with the robot arm 110 and the imaging system 106. For example, the controller 206 may be implemented as a device mounted on or integrated within a stage to which the robot arm 110 and / or the imaging system 106 are attached. Alternatively, the controller 206 may be located away from the robot inspection platform 100.

[0029] In any case, the controller 206 may be coupled to the robot arm 110 and / or imaging system 106 via a wired link, a wireless link, or any suitable combination thereof. Thus, in Figure 3, links 210 and / or 212 may each represent one or more wired links and / or wireless links to facilitate communication between the controller 206 and one or both of the robot arm 110 and the imaging system 106. Although Figure 3 shows two separate links 210 and 212, it is understood that the controller 206 may communicate with the robot arm 110 and the imaging system 106 via any suitable number of links, such as a single common link or bus.

[0030] To facilitate communication with and control of the robot arm 110 and the imaging system 106, the controller 206 may include a processing unit 220, a communication unit 222, and a memory unit 224. The processing unit 220 may be implemented as any suitable type and / or number of processors, for example, the host processor of the controller 206. To provide further examples, the processing unit 220 may be implemented as an application-specific integrated circuit (ASIC), embedded processor, central processing unit (CPU), etc., associated with the controller 206. Although not shown in Figure 3, the processing unit 220 may be coupled to the communication unit 222 and / or the memory unit 224 via one or more wired and / or wireless interconnections, for example, any suitable number of data and / or address buses (and / or otherwise configured to communicate with, control, operate with, and / or influence this operation).

[0031] For example, the processing unit 220 may be configured to retrieve, process, and / or analyze data stored in the memory unit 224 (e.g., images captured by one or more imagers of the imaging system 106), to store such data in the memory unit 224, to replace the data stored in the memory unit 224, and to control various functions associated with the robot arm 110 (e.g., any of the robot arm functions described herein) and / or the imaging system 106 (e.g., any of the imaging functions described herein).

[0032] The communication unit 222 may be configured to support any suitable number and / or types of communication protocols to facilitate communication between the controller 206 and one or both of the robot arm 110 and the imaging system 106. The communication unit 222 may also be configured to facilitate the exchange of any suitable type of information (e.g., via links 210 and / or link 212) between the controller 206 and one or both of the robot arm 110 and the imaging system 106, and may be implemented as any suitable combination of hardware and / or software to facilitate such functionality. For example, the communication unit 222 may implement any number of wired and / or wireless transceivers, modems, ports, input / output interfaces, connectors, antennas, etc. Although not shown in Figure 3, the robot arm 110 and / or the imaging system 106 may include similar types of communication units to enable communication via links 210 and / or link 212.

[0033] According to various embodiments, the memory unit 224 may be one or more computer-readable non-temporary storage devices, including any suitable volatile memory (e.g., random access memory (RAM)), non-volatile memory (e.g., battery-backed RAM, FLASH, etc.), or a combination thereof. The memory unit 224 may be configured to store instructions that can be executed by the processing unit 220. These instructions may include machine-readable instructions that, when executed by the processing unit 220, cause the processing unit 220 to perform various actions described herein. While various functions of the controller 206 are described herein in terms of the execution of instructions stored in the memory unit 224 by the processing unit 220, it will be understood that equivalent functions may instead be implemented using only hardware components (e.g., hardware processors). The memory unit 224 may also be configured to store any other suitable data used with the robotic inspection platform 100, such as images captured by the imager of the imaging system 106, data indicating the attributes of identified samples and / or containers, etc.

[0034] The control module 230 is a set of instructions in the memory unit 224 that, when executed by the processing unit 220, causes the processing unit 220 to perform various actions in accordance with suitable embodiments described herein. In one embodiment, the control module 230, when executed by the processing unit 220, includes instructions that cause the processing unit 220 to send one or more commands to the robot arm 110 to control the operation (e.g., motion) of the robot arm 110, for example, as described above in relation to Figure 1. Furthermore, the control module 230 may also include instructions that, when executed by the processing unit 220, cause the processing unit 220 to send one or more commands to the imaging system 106 to control the operation of the imaging system 106, for example, as described above in relation to Figure 1.

[0035] For example, the processing unit 220 may send a command to the robot arm 110 via the communication unit 222, which causes the robot arm 110 to sequentially remove the containers 104 from the plate 102, place each container 104 within the field of view of one or more imagers of the imaging system 106, and manipulate each container 104 to take on multiple orientations while within the imager's field of view. Alternatively, the processing unit 220 may send a command to the imaging system 106 via the communication unit 222, which causes the imager (or multiple imagers) to move relative to a stationary container (for example, to rotate around the container while facing it so that the container remains within the field of view). In either case, over the same time frame, the processing unit 220 may send a command to the imaging system 106 via the communication unit 222, which causes the imager to capture images of each container 104 while it is in each of the relative orientations described above, provide these images to the controller 206, and store them in the memory unit 224. In some embodiments, the processing unit 220 also sends commands to the imaging system 106 via the communication unit 222, which cause the light source of the imaging system 106 to be turned on and off as appropriate (for example, so that each container 104 is illuminated when an image is captured), and / or to set other parameters of such light sources (e.g., intensity level, wavelength, etc.).

[0036] The defective product identification module 232, when executed by the processing unit 220, is a set of instructions in the memory unit 224 that cause the processing unit 220 to perform various actions in accordance with suitable embodiments described herein. In one embodiment, the defective product identification module 232, when executed by the processing unit 220, includes instructions that cause the processing unit 220 to process container / sample images received from the imaging system 106 using the pattern recognition module 240.

[0037] The pattern recognition module 240 may process images (for example, one per container, one per orientation, or multiple per orientation) using any suitable pattern recognition model and determine one or more attributes of the containers and / or samples corresponding to these images. In some embodiments, the pattern recognition module 240 implements a machine learning model trained using supervised learning, as further described below. Generally, the pattern recognition model may use any suitable technique, such as parametric or nonparametric classification algorithms, neural networks, support vector machines, etc. The pattern recognition module 240 is trained or designed to identify specific attributes, such as any of the attributes described herein (e.g., objects, a particular type of object, structural defects, etc.).

[0038] In some embodiments, the pattern recognition module 240 implements a trained machine learning model that accepts an input (e.g., one or more images of a container), the input passes through the trained model once, and generates an output (e.g., one or more attribute classifications of the container and / or the sample inside the container) in a process called "inference". Such inference can be performed very quickly by utilizing any suitable technology or advancement with respect to hardware or other processing units (e.g., FPGA, ASIC, graphics processing unit (GPU), etc.), resulting in very fast and efficient inspection of the container and / or sample.

[0039] In some embodiments, the pattern recognition module 240 outputs not only classifications / labels / categories indicating specific container and / or sample attributes, but also confidence scores that reflect the probability of the classification being correct. For example, the pattern recognition module 240 may output a value [particle, 0.67222] indicating a 67.222% probability that a unique particle is present in the sample, a value [particle A, 0.853] indicating an 85.3% probability that the sample contains a unique particle of type A, or a value [3, particle A, 0.510] indicating a 51% probability that the sample contains exactly three particles of type A.

[0040] In some embodiments, the rejection module 232 is configured to determine additional attributes not output by the pattern recognition module 240. For example, the rejection module 232 may determine / measure the size of an object classified by the pattern recognition module 240, and / or determine / count the number of objects in a particular class, etc. Whether or not the rejection module 232 determines any additional attributes, the rejection module 232 may use the determined attributes (and, if possible, the corresponding confidence scores as well) to determine whether the container should be rejected, i.e., whether the container and / or sample does not meet one or more criteria.

[0041] For example, the rejection module 232 may reject a container holding a sample in which the pattern recognition module 240 has identified a threshold number of particles and / or bubbles, a threshold number of specific types of particles and / or bubbles, or a threshold number of specific types of bubbles having at least a threshold size in the corresponding image. In some embodiments, the rejection module 232 calculates a score which is a mathematical function of the attribute and the confidence score. If the score is above (or below) a predetermined threshold, the container is rejected. After the rejection module 232 has decided whether a given container should be rejected, the processing unit 220 may send a command to the robot arm 110 via the communication unit 222 to place the container into the rejection box 120 (if rejected) or return it to plate 102 or another appropriate location (if not rejected).

[0042] In some embodiments, as described above, the robot inspection platform 100 may be used not only to analyze samples (and / or containers themselves) using a trained machine learning model, but also to facilitate the training of the model. In such embodiments, the processing unit 220 may perform a process in which images of training containers (i.e., containers holding training samples or empty training containers) are captured and presented to a human analyst for manual labeling, and the labeled images are used in the supervised learning process. In particular, the processing unit 220 may send commands to the robot arm 110 via the communication unit 222, which cause the robot arm 110 to sequentially retrieve training containers from a plate (e.g., similar to plate 102) or other suitable source area, place each training container within the field of view of the imager of the imaging system 106, and manipulate each training container to take multiple orientations while within the field of view of the imager. Over the same time frame, the processing unit 220 may also send commands to the imaging system 106 via the communication unit 222, which cause the imager of the imaging system 106 to capture images of each training container while the container (or imager) is in several different orientations, and to provide these training images to the controller 206 for storage in the memory unit 224. In some embodiments, the processing unit 220 may also send commands to the imaging system 106 via the communication unit 222, which cause the light source of the imaging system 106 to be turned on and off as appropriate (for example, so that each training container is illuminated when a training image is captured), and / or to set other parameters of such light source (e.g., intensity level, wavelength, etc.).

[0043] In general, the processing unit 220 may operate the robot inspection platform 100 in the same manner as during runtime. For example, the training sample container (or imager) may be operated to take the same orientation as it will be used later during runtime, as described above. If the light source has computer-controllable parameters (e.g., intensity, wavelength, etc.), the training process may also include the processing unit 220 sending commands to ensure that the same light source parameters as those used later during runtime are present during training. With respect to any manually adjustable parameters or configurations of the light source, it is generally important that the parameters / configurations are the same or at least very similar during both training and runtime.

[0044] A human analyst may label training images using various methods depending on the embodiment. For example, the processing unit 220 may store training images in the memory unit 224 and generate a graphical user interface (GUI) that presents the stored images with respect to a given training container. The processing unit 220 may also provide various interactive functions to assist the analyst's labeling process. In one such embodiment, the analyst may drag or draw a rectangle or other polygon around an object or other attribute in the image and type (or select from a menu, etc.) the classification of the enclosed attribute. Referring to Figure 2, for example, an analyst looking at an image of the shown syringe 150 may draw a rectangle around particle 160F and type or select the type of particle that they believe accurately depicts particle 160F. In another example, the analyst may draw a rectangle around bubble 160C and type or select "bubble" (or a specific type of bubble) as a label, etc.

[0045] Figure 4 is a flowchart illustrating an exemplary method 300, according to one embodiment, for performing inspection of a container (i.e., a sample inside the container and / or the container itself) using pattern recognition. Method 300 may be performed entirely or partially by, for example, a controller 206, a robotic arm 110, and / or an imaging system 106. In some embodiments, the entirety of Method 300 is performed by a processing unit 220 of the controller 206 when instructions stored in a memory unit 224 are executed.

[0046] In Method 300, a robotic arm is instructed to retrieve a container using the robotic arm's end effector (block 302). The robotic arm may be similar to, for example, the robotic arm 110 in Figures 1 and 3, and the end effector may be similar to the end effector 112 in Figure 1. Depending on the embodiment, the container may be a syringe, vial, cartridge, or any other type of container that may or may not contain a sample (e.g., a fluid or lyophilized product). Block 302 may be implemented, for example, by generating one or more command signals and transmitting the signals to the robotic arm (e.g., a slave controller in the robotic arm).

[0047] The robotic arm is instructed to manipulate the container using an end effector (block 304) so ​​that the container is sequentially positioned in multiple orientations while it is within the field of view of an imager (e.g., the imager of imaging system 106 in Figure 1). In some embodiments, the container is positioned simultaneously within the field of view of one or more additional imagers (e.g., a total of three imagers, each 120 degrees apart in a two-dimensional plane). Block 304 may be implemented, for example, by generating one or more command signals and transmitting the signals to the robotic arm (e.g., a slave controller within the robotic arm).

[0048] The imager is made to capture multiple images (block 306), each image being captured while the container is in one of the orientations. The orientations may include, for example, different rotational positions with respect to the longitudinal axis of the container (e.g., every 20 degrees or every 90 degrees of a full circle). Block 306 may be performed, for example, by generating one or more command signals and sending the signals to the imager (e.g., a slave controller in the imager). Block 306 may further include receiving the images captured by the imager and storing these images in memory (e.g., memory unit 224 in Figure 3).

[0049] In another embodiment, block 304 may instead (or further) include moving the imager (or a plurality of imagers) relative to the container such that the relative orientation between the container and the imager changes without necessarily requiring any additional movement of the container (e.g., rotation) via the end effector of the robot arm. For example, the container may remain stationary once it is initially placed in a predetermined position by the robot arm, and the imager may move along a circular path around the container (e.g., a full circle in 20-degree increments). In such an embodiment, block 306 may include causing the imager to capture one or more images at each position of the imager.

[0050] One or more attributes of the container and / or sample are determined by analyzing the image captured in block 306 using a pattern recognition model (block 308). The attributes may also be determined by analyzing the container and / or sample using a machine learning model, for example, a model trained to classify the attributes of the container and / or sample (e.g., using supervised learning). The attributes may include the presence of general objects, the presence of specific types of objects, dirt or certain types of dirt, cracks or deformations of the container, and / or other attributes.

[0051] Based on the attributes determined in block 308, determine whether the container and / or sample meet one or more criteria (block 310). The criteria may include, for example, at least one criterion depending on the type of object (e.g., whether the sample contains particles, foam, a specific type of particle or foam, etc.). One or more criteria may also take into account confidence levels / scores associated with one or more attributes.

[0052] Based on whether one or more criteria are met, the robot arm is instructed to place the container into an area reserved for rejected containers and / or samples (e.g., the rejected container box 120 in Figure 1), or into an area reserved for containers and / or samples that did not fail (e.g., the plate 102 in Figure 1) (block 312). Block 312 may be performed, for example, by generating one or more command signals and sending the signals to the robot arm (e.g., a slave controller in the robot arm).

[0053] In some embodiments, Method 300 includes one or more additional blocks not shown in Figure 4. In some embodiments in which a machine learning model is used in block 306, for example, Method 300 may include several additional blocks corresponding to training the machine learning model before block 302. For example, Method 300 may include an additional block in which a robotic arm sequentially retrieves a plurality of training containers using an end effector. With respect to each retrieved training container, Method may include instructing the robotic arm to manipulate the container using the end effector so that the container is sequentially positioned in a plurality of orientations while the container is in the imager's field of view, causing the imager to capture each training image (each of the training images is captured while the training container is in each of the orientations), and presenting each image to a user via a display to facilitate manual labeling of attributes relating to the training container and / or the sample in the training container. These additional blocks may include, for example, generating command signals and transmitting signals to appropriate destinations (e.g., a slave controller on the robotic arm and a slave controller on the imager).

[0054] Figure 5 is a flowchart of an exemplary method 350 for performing inspection of a series of containers according to one embodiment. Method 350 may be performed entirely or partially by, for example, a controller 206 (e.g., a processing unit 220), a robotic arm 110, and / or an imaging system 106. As further described below, method 350 may be used in conjunction with method 300 of Figure 4.

[0055] Method 350 may begin by capturing images of containers in different orientations (block 352) and processing the captured images to determine the attributes of the containers (and / or the samples inside the containers) and whether one or more criteria are met (block 354). Containers may be classified as unacceptable if one or more criteria are not met, or as acceptable if one or more criteria are met. For a given container, for example, block 352 may correspond to blocks 302-306 of Method 300, and block 354 may correspond to blocks 308 and 310 of Method 300.

[0056] Once a given container has been analyzed and classified as either a reject or a pass, method 350 may include determining whether all containers selected for analysis have already been analyzed (block 356). This determination may be made, for example, with respect to a given type of plate having a known number of containers (e.g., 24 or 96), by tracking the total number of containers analyzed and determining whether this number is less than or equal to the total number of containers. Alternatively, the determination may be made, for example, by having a user place any appropriate number of containers (which may be less than the total number of containers in the plate) for analysis, tracking the total number of containers analyzed, and determining whether this number is less than or equal to the number of test containers placed.

[0057] In any case, if further containers need to be analyzed, method 350 continues by removing the next container (block 358). Once the next container is properly positioned, method 350 may repeat the process by capturing an image of the container and / or the sample inside the container (block 352). Conversely, if the analyzed container is the last container to be analyzed, method 350 terminates. Upon termination (or in parallel with the iteration of block 354), method 350 may include storing or printing an analysis report for each container (for example, to the memory unit 224 as shown in Figure 3). Once this report is complete, the user may view the report and / or start the process with respect to a new plate.

[0058] Some of the figures described herein show exemplary block diagrams having one or more functional components. Such block diagrams are for illustrative purposes only, and it will be understood that the devices described and illustrated may have more, fewer, or different components than those illustrated. Furthermore, in various embodiments, components (and the functions provided by each component) may be associated with or integrated as part of any suitable component.

[0059] Embodiments of this disclosure relate to non-temporary computer-readable storage media having computer code for performing various operations performed by a computer. As used herein, the term “computer-readable storage media” includes any medium capable of storing or encoding a set of instructions or computer code for performing the operations, techniques, and methods described herein. The medium and computer code may be specifically designed and constructed for embodiments of this disclosure, or may be of a type well known and available to those skilled in the art of computer software technology. Examples of computer-readable storage media include, but are not limited to, magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and holographic devices; magneto-optical media such as optical discs; and hardware devices specifically configured to store and execute program code, such as ASICs, programmable logic devices ("PLDs"), and ROM and RAM devices.

[0060] Examples of computer code include machine code, such as that generated by a compiler, and files containing higher-level code that is executed by a computer using an interpreter or compiler. For example, embodiments of the disclosure may be implemented using Java, C++, or other object-oriented programming languages ​​and development tools. Further examples of computer code include encrypted code and compressed code. Furthermore, embodiments of the disclosure may be downloaded as a computer program product, which may be transferred from a remote computer (e.g., a server computer) to a requesting computer (e.g., a client computer or a different server computer) via a transmission channel. Another embodiment of the disclosure may be implemented in hardwired circuitry instead of, or in conjunction with, machine-executable software instructions.

[0061] As used herein, the singular terms “a,” “an,” and “the” may refer to multiple objects unless otherwise specified in the context.

[0062] As used herein, relative terms such as "above," "below," "up," "left," "right," "down," "top," "bottom," "vertical," "horizontal," "side," "higher," "lower," "upper," "over," "under," "inner," "interior," "outer," "exterior," "front," "back," "upwardly," "downwardly," "vertically," "lateral," and "laterally" refer to the orientation of a series of components relative to each other. This orientation follows the drawings but is not required during manufacture or use.

[0063] As used herein, the terms “connect,” “connected,” and “connection” refer to an operational combination or linkage. Connected components may be connected to one another directly or indirectly, for example, through another set of components.

[0064] As used herein, the terms “approximately,” “substantially,” “substantial,” and “about” are used to describe and explain small variations. When used with events or situations, these terms may mean that the event or situation occurs precisely, and that it occurs close approximationally. For example, when used with numbers, these terms may mean a range of variation of that number of ±10%, such as ±5%, ±4%, ±3%, ±2%, ±1%, ±0.5%, ±0.1%, or ±0.05%. For example, two numbers can be considered “substantially” the same if the difference between them is within ±10% of the mean of those numbers, such as ±5%, ±4%, ±3%, ±2%, ±1%, ±0.5%, ±0.1%, or ±0.05%.

[0065] Furthermore, quantities, ratios, and other numerical values ​​may be expressed in the form of ranges in this specification. Such range forms are used for convenience and simplification and include numerical values ​​that are explicitly stated as limits to the range, but should be understood to be flexibly interpreted to include any individual numerical values ​​or subranges contained within that range, as if each numerical value and subrange were explicitly stated.

[0066] While this disclosure has been described and illustrated with reference to specific embodiments thereof, these descriptions and illustrations are not intended to limit the disclosure. Those skilled in the art will understand that various modifications may be made and equivalents may be substituted without departing from the true intent and scope of the disclosure as defined in the appended claims. The drawings may not be drawn to scale. Differences may exist between the artistic renditions in this disclosure and the actual apparatus due to manufacturing processes and tolerances. Other embodiments of this disclosure that are not specifically illustrated may exist. The specification and drawings should be considered illustrative rather than restrictive. Modifications may be made to suit specific circumstances, materials, substance compositions, methods, or processes. All such modifications shall be included in the claims appended to this specification. While the methods disclosed herein have been described with reference to specific operations performed in a particular order, it will be understood that these operations may be combined, subdivided, or rearranged to form equivalent methods without departing from the teachings of this disclosure. Therefore, unless expressly stated herein, the order and grouping of operations are not limited to this disclosure.

Claims

1. A robotic arm comprising an end effector and multiple articulated segments, configured to manipulate a container with at least four degrees of freedom of motion, It is a controller, The robot arm is made to remove the container using the end effector. While the container is within the imager's field of view, the robot arm is made to manipulate the container using the end effector. By analyzing the image of the container captured by the imager using a pattern recognition model, one or more attributes of the container and the sample inside the container are determined. Based on one or more attributes of the container and the sample, either (i) cause the robot arm to place the container into the area reserved for unacceptable containers and / or samples, or (ii) cause the robot arm to place the container into the area reserved for containers and / or samples that did not fail the inspection. A controller and, A robotic inspection platform, including...

2. The robotic inspection platform according to claim 1, wherein the controller is configured to determine one or more attributes of the container and the sample by analyzing the image using a machine learning model, and the machine learning model is trained to classify the attributes of the container and / or the sample.

3. The controller comprises at least, The robot arm is made to sequentially retrieve multiple training containers using the end effector, With respect to each of the removed training containers, (i) the robot arm is made to manipulate the training container using the end effector so that the training container is sequentially positioned in a plurality of orientations while the training container is within the field of view of the imager; (ii) the imager is made to capture each of the plurality of training images, each of which is captured while the training container is in one of the plurality of orientations; and (iii) each of the plurality of training images is presented to the user via a display to facilitate manual labeling of the attributes of the training container and / or the sample in the training container. The robot inspection platform according to claim 2, configured to facilitate the training of the machine learning model.

4. The robot inspection platform according to any one of claims 1 to 3, wherein the controller is configured to distinguish between different types of objects by analyzing images using the pattern recognition model.

5. The robotic inspection platform according to claim 4, wherein the different types of objects include either or both bubbles and particles.

6. The robotic inspection platform according to claim 4 or 5, wherein the different types of objects include one or both of (i) a specific type of foam and (ii) a specific type of particle.

7. The robotic inspection platform according to any one of claims 4 to 6, wherein one or more criteria based on one or more attributes of the container and the sample include at least one criterion depending on the type of object.

8. The robot inspection platform according to any one of claims 1 to 7, wherein the robot arm is configured to sequentially arrange the containers in a plurality of orientations, including a plurality of rotations around the longitudinal axis of the containers.

9. The robotic inspection platform according to any one of claims 1 to 8, wherein the area reserved for unacceptable containers and / or samples is a box.

10. The robot inspection platform according to any one of claims 1 to 9, wherein the controller is configured to cause the robot arm to remove the container from the plate, and the area reserved for containers and / or samples that did not fail the inspection is an area within the plate.

11. The robotic arm is configured to manipulate the container with six degrees of freedom of motion. The robot inspection platform according to any one of claims 1 to 10, wherein the container has a first roll angle and a first pitch angle in a first orientation of a plurality of orientations, and the container has (i) a second roll angle different from the first roll angle or (ii) a second pitch angle different from the first pitch angle in a second orientation of the plurality of orientations.

12. A method for performing container inspection using pattern recognition, The robot arm is made to remove a container using the end effector of the robot arm, wherein the robot arm is configured to manipulate the container with at least four degrees of freedom of motion. While the container is within the imager's field of view, the robot arm is instructed to manipulate the container using the end effector. The process involves determining one or more attributes of the container and the samples within the container by analyzing the image of the container captured by the imager using a pattern recognition model, Based on one or more attributes of the container and the sample, either (i) cause the robot arm to place the container in an area reserved for unacceptable containers and / or samples, or (ii) cause the robot arm to place the container in an area reserved for containers and / or samples that did not fail the inspection. Methods that include...

13. Determining one or more attributes of the container and the sample by analyzing the image using a pattern recognition model is: Determining one or more attributes of the container and the sample by analyzing the image using a machine learning model, wherein the machine learning model is trained to classify the attributes of the container and / or the sample. The method according to claim 12, including the method described in claim 12.

14. at least, The robot arm is made to sequentially retrieve multiple training containers using the end effector, With respect to each of the removed training containers, (i) causing the robotic arm to manipulate the training container using the end effector so that the training container is sequentially positioned in a plurality of orientations while the training container is within the field of view of the imager; (ii) causing the imager to capture each of the plurality of training images, each of which is captured while the training container is in one of the plurality of orientations; and (iii) presenting each of the plurality of training images to the user via a display in order to facilitate manual labeling of the attributes relating to the training container and / or the sample in the training container. The method according to claim 13, further comprising facilitating the training of the machine learning model.

15. The method according to any one of claims 12 to 14, wherein determining one or more attributes of the container and the sample by analyzing the image using the pattern recognition model includes distinguishing between different types of objects by analyzing the image using the pattern recognition model, and one or more criteria based on the one or more attributes of the container and the sample include at least one criterion depending on the type of object.

16. The robot arm is to sequentially position the containers in multiple orientations, including multiple rotations around the longitudinal axis of the containers. The method according to any one of claims 12 to 15, further comprising:

17. The robotic arm is configured to manipulate the container with six degrees of freedom of motion. The method according to any one of claims 12 to 16, wherein the container has a first roll angle and a first pitch angle in a first orientation of a plurality of orientations, and the container has (i) a second roll angle different from the first roll angle or (ii) a second pitch angle different from the first pitch angle in a second orientation of the plurality of orientations.

18. A method for performing container inspection using pattern recognition, The robot arm is made to remove a container using the end effector of the robot arm, wherein the robot arm is configured to manipulate the container with at least four degrees of freedom of motion. The imager is arranged sequentially in multiple orientations while the container is within the imager's field of view. The process involves determining one or more attributes of the container and sample by analyzing the image of the container captured by the imager using a pattern recognition model, Based on one or more attributes of the container and the sample, either (i) cause the robot arm to place the container in an area reserved for unacceptable containers and / or samples, or (ii) cause the robot arm to place the container in an area reserved for containers and / or samples that did not fail the inspection. Methods that include...

19. Determining one or more attributes of the container and the sample by analyzing the image using a pattern recognition model is: Determining one or more attributes of the container and the sample by analyzing the image using a machine learning model, wherein the machine learning model is trained to classify the attributes of the container and / or the sample. The method according to claim 18, including the method described in claim 18.

20. at least, The robot arm is made to sequentially retrieve multiple training containers using the end effector, With respect to each of the removed training containers, (i) sequentially positioning the imager in the plurality of orientations while the training container is within the imager's field of view; (ii) having the imager capture each of the plurality of training images, each of which is captured while the imager is in each of the plurality of orientations; and (iii) presenting each of the plurality of training images to the user via a display in order to facilitate manual labeling of the attributes relating to the training container and / or the sample in the training container. The method according to claim 19, further comprising facilitating the training of the machine learning model.

Citation Information

Patent Citations

  • Visible foreign matter and bubble classification recognition detection method for medical (250ml)

    CN105869154A

  • Surface condition testing method and surface condition testing device

    JP2007240434A

  • Method and Apparatus for Detecting Foreign Objects or Defects in Multiple Filled Containers

    JP2008510971A

  • Visual inspection apparatus

    JP2009174943A

  • Drug verification and supply

    JP2013529095A