Targeted Application of Deep Learning to Automated Visual Inspection Machines

Abstract

In a method for increasing accuracy and efficiency in automated visual inspection of containers, a container containing a sample is oriented such that a line scan camera obtains a profile image of an edge of a stopper of the container. While the container is being spun, multiple images of the edge of the stopper are captured by a first line scan camera, each image of the multiple images corresponding to a different rotational position of the container. A two-dimensional image of the edge of the stopper is generated based on at least the multiple images, and pixels of the two-dimensional image are processed by one or more processors executing an inference model including a trained neural network to generate output data indicative of the likelihood that the sample contains a defect.

Description

Technical Field

[0001] This application generally relates to an automated visual inspection (AVI) system for pharmaceuticals or other products, and more particularly to techniques for detecting and differentiating particles and other objects (e.g., air bubbles) within containers filled with a sample (e.g., a solution).

Background Art

[0002] In certain situations, such as in quality control procedures for manufactured pharmaceuticals, it is necessary to examine whether a sample (e.g., a container / vessel such as a syringe or vial, and / or the contents such as a liquid or lyophilized pharmaceutical) has any defects. The acceptability of a particular sample under applicable quality criteria may depend on measurement criteria such as, for example, the type and / or size of container defects (e.g., chips or cracks), or the type, number, and / or size of unwanted particles (e.g., fibers) in the pharmaceutical. If a sample has non - acceptable measurement criteria, it may be rejected and / or discarded.

[0003] To handle the volumes typically associated with commercial pharmaceutical production, defect inspection tasks are increasingly being automated. However, the automated detection of particulate matter in solution presents particular challenges in the pharmaceutical industry. Achieving high detection accuracy is generally difficult, and is made more difficult by higher - viscosity solutions that impede the movement of particles that might otherwise indicate a particle type. In the case of protein - based products with formulations that release gases promoting bubble formation, conventional particle detection techniques can result in particularly high rates of false rejections. For example, with such techniques, it can be difficult to distinguish these air bubbles (which may adhere to the container) from heavy particles that tend to sediment / settle against a portion of the container (e.g., against the plunger of a syringe filled with solution).

[0004] Furthermore, the specialized equipment used to support automated defect inspection has become extremely large, complex, and expensive. A single commercial line of equipment may include numerous different AVI stations, each handling a different specific inspection task. As just one example, the Bosch® Automated Inspection Machine (AIM) 5023 commercial line of equipment used in the completion inspection stage of drug-filled syringes includes 16 general inspection tasks and 14 separate visual inspection stations equipped with numerous cameras and other sensors. Overall, such equipment may be designed to detect a wide range of defects, including defects in container integrity such as large cracks or container closure, cosmetic defects such as scratches or stains on the container surface, and defects related to the drug itself, such as color of the liquid or the presence of foreign particles. However, due to the aforementioned challenges related to particle detection and characterization, such equipment may require redundancy between AVI stations. For example, in the case of the Bosch® AIM 5023 line instrument, the relatively low performance of the “stopper edge” inspection station (for detecting and distinguishing heavy particles stationary on the dome of the syringe plunger) necessitates performing particle inspection at a separate “stopper top” AVI station with an additional camera to make the overall level of particle inspection accuracy acceptable. This increases the complexity and cost of the instrument and / or requires the “stopper top” AVI station to be adapted to perform multiple inspection tasks rather than being optimized for a single task (e.g., detecting defects in the stopper itself). [Overview of the project] [Means for solving the problem]

[0005] Embodiments described herein relate to systems and methods for applying deep learning to certain types of AVI stations (e.g., in commercial line equipment which may include multiple AVI stations) to significantly improve accuracy through synergistic effects (e.g., far fewer false rejections and / or false positives). In addition, or instead, the described systems and methods may allow favorable modifications to other AVI stations (e.g., in the same commercial line equipment), such as allowing other AVI stations to dedicate themselves to other tasks and / or eliminating other AVI stations altogether.

[0006] Specifically, deep learning is applied to an AVI station utilizing one or more line-scan cameras (e.g., CMOS line-scan cameras) to detect and distinguish objects stationary or placed near the edge of the stopper of a container containing a sample (e.g., liquid solution pharmaceuticals) (e.g., glass and / or other particles relative to gas-filled bubbles). For example, the AVI station may utilize the line-scan camera to detect and distinguish objects placed on or near the surface of the syringe plunger dome in contact with the liquid sample in the syringe. The line-scan camera may capture multiple line images as the AVI station rotates / spins the container at least one full turn (360 degrees), after which a processing device or a component within (or communicatively coupled to) the AVI station generates a two-dimensional image from the multiple line images.

[0007] The AVI station or an external processing component provides a trained neural network with pixel values ​​(e.g., normalized pixel intensity values) from a 2D image. This allows the system to infer whether a container sample is unacceptable (e.g., whether particles in the imaged area contain unacceptable numbers, sizes, and / or types). The neural network may be trained using supervised learning techniques with a wide range of 2D images of samples known (and labeled) to have acceptable or unacceptable numbers, types, sizes, etc., of particles and / or gas-filled bubbles. The selection and classification of images used to train the neural network are crucial to its performance in the inference phase. Furthermore, to avoid accepting defective units, unexpected situations should be anticipated and included in the training images. Importantly, the trained neural network, or a larger inference model containing the neural network, may be "locked" before certification so that the model cannot be modified (e.g., further trained) without recertification. Preferably, acceptance criteria should be established and pre-approved to ensure that the system performs as well as or better than manual visual inspection.

[0008] If an AVI station (or a communicationally coupled processing device) indicates that a sample is defective, the AVI station, or commercial line equipment including an AVI station, may physically transport the container / sample to a rejection area, where the sample may be discarded / destroyed or transferred for further inspection (e.g., manual inspection). Depending on the embodiment, the container / sample may be transported directly to a discharge / rejection area (e.g., a bin), or it may first pass through one or more other AVI stations. If the inference model does not indicate that a sample is defective, the AVI station or commercial line equipment may transport the container / sample directly to an area designated for accepted products, or to the next AVI station (e.g., one or more AVI stations designed to detect defects in other types of samples and / or containers) for further inspection.

[0009] Those skilled in the art will understand that the figures described herein are included for illustrative purposes only and do not limit the disclosure. The drawings are not necessarily to scale and instead focus on illustrating the principles of the disclosure. In some cases, various aspects of the described embodiments may be exaggerated or enlarged to facilitate understanding of the described embodiments. In the drawings, similar reference numerals appearing throughout the drawings refer to components that are generally functionally and / or structurally similar. [Brief explanation of the drawing]

[0010] [Figure 1] This is a simplified block diagram of an exemplary line device in which the imaging and deep learning techniques described herein may be implemented. [Figure 2] This is a simplified depiction of an AVI station within conventional commercial line equipment. [Figure 3A] An example container is shown where the edge of the container's stopper is imaged using a line scan camera. [Figure 3B] An example container is shown where the edge of the container's stopper is imaged using a line scan camera. [Figure 4] This shows an exemplary two-dimensional stopper edge image that can be generated from line images captured by a line scan camera. [Figure 5] Figure 4 shows an exemplary neural network that can be used to infer the acceptability or unacceptability of a sample based on an image, such as a two-dimensional image. [Figure 6] This document illustrates the stages of an exemplary development and certification process for implementing deep learning using AVI stations. [Figure 7] This shows the results of a proof of concept obtained when deep learning was used on a specific AVI station. [Figure 8] This is a flowchart illustrating an exemplary method for improving the accuracy and efficiency of automated visual inspection of containers. [Modes for carrying out the invention]

[0011] The various concepts introduced above and described in more detail below can be implemented in one of many ways, and the concepts described are not limited to any particular implementation method. Examples of implementations are provided for illustrative purposes only.

[0012] Figure 1 is a simplified block diagram of an exemplary line equipment 100 in which the technology described herein may be implemented. Line equipment 100 may be any production-grade equipment having, for example, N (N≧1) AVI stations 110-1 to 110-N (collectively also referred to as AVI stations 110). To provide a more specific example, line equipment 100 may be a modified version of the Bosch® Automated Inspection Machine (AIM) 5023 commercial line equipment, which is discussed below with reference to Figure 2. Each of the AVI stations 110 may be responsible for capturing images used for inspection of various forms of containers (e.g., syringes, vials, etc.) and / or samples within the containers (e.g., liquid solution pharmaceuticals). For example, a first AVI station 110-1 may capture a top view image of a syringe, vial, or other container to inspect for cracks or tips, and a second AVI station 110-2 (not shown in Figure 1) may capture a side view image to inspect the entire sample in the container for the presence or absence of foreign particles.

[0013] Figure 1 also shows a simplified block diagram illustrating the general components of the i-th AVI station 110-i, where i may be any integer from 1 to N. Specifically, the AVI station 110-i is configured to automatically and visually inspect a sample (the contents of a container) in the area where the sample meets / contacts the edge of the container's stopper. The stopper may be, for example, a syringe plunger, or a cap or plug that seals the opening of a vial. To perform this inspection, the AVI station 110-i includes an imaging system 112, an illumination system 114, and sample positioning hardware 116. Other AVI stations 110 (if any) may have generally similar types of components (e.g., imaging system, illumination system, and sample positioning hardware), but it should be understood that in some cases, there may be different types and configurations of components appropriate for each given station 110.

[0014] The imaging system 112 includes at least one line scan camera and possibly associated optical components (e.g., additional lenses, mirrors, filters, etc.) to capture line images of each sample (pharmaceutical). Each of the line scan cameras may be, for example, a CMOS line scan camera. For simplicity, most of the following description will refer to a single line scan camera only. However, it should be understood that multiple line scan cameras may be used. For example, to improve throughput, each of two line scan cameras may image different containers / samples simultaneously in parallel.

[0015] The illumination system 114 includes one or more illumination devices to illuminate each sample while the sample is being imaged by the line scan camera. The illumination devices may include one or more light-emitting diodes (LEDs), such as an LED array arranged as a backlight panel.

[0016] The sample positioning hardware 116 may include any hardware for holding (or otherwise supporting) and moving containers for the AVI station 110-i. In the embodiment of Figure 1, the sample positioning hardware 116 includes at least a transport means 117 for orienting each container so that the line scan camera of the imaging system 112 obtains a profile image of the container's stopper edge, and a spinning means 118 for spinning each container (e.g., rotating it around the central axis of the container) while the line scan camera captures a line image. The transport means 117 may include an electric rotary table, a star wheel or carousel, a robotic arm, and / or any other suitable mechanism for orienting (e.g., moving and positioning) each container. The spinning means 118 may include, for example, an electric spinning mechanism (e.g., a component of Bosch® AIM 5023 that provides a “direct spin” function for syringes, as discussed below with reference to Figure 2). As will be further explained below, after the transport means 117 has appropriately positioned / orient the given container, the spinning means 118 spins the container so that the line scan camera can capture a line image that comprehensively covers the entire 360-degree view of the stopper in the area where the stopper contacts the sample.

[0017] In some embodiments, the sample positioning hardware 116 also includes hardware for inverting each container (e.g., to ensure that the stopper is positioned below the sample when imaging is being performed, to increase the likelihood that heavy particles will rest directly on the stopper), and / or for stirring the sample contained in each container. In other embodiments, specific aspects of properly orienting each container (e.g., container inversion) occur at the initial AVI station 110, within the initial AVI station 110, or before handling by line equipment 100, etc. Various exemplary orientations of the line scan camera relative to the containers / samples when the line scan camera captures images of spinning samples are described below with reference to Figures 3A and 3B.

[0018] The line device 100 also includes one or more processors 120 and memory 122. Each of the processors 120 may be a programmable microprocessor that executes software instructions stored in memory 122 to perform some or all of the software control functions of the line device 100 as described herein. Alternatively or in addition, one or more of the processors 120 may be other types of processors (e.g., application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc.), and some of the functions of the processors 120 described herein may be implemented in hardware instead. Memory 122 may include one or more volatile and / or non-volatile memories. Memory 122 may include one or more suitable memory types, such as read-only memory (ROM), random access memory (RAM), flash memory, solid-state drives (SSDs), and hard disk drives (HDDs). Collectively, memory 122 may store one or more software applications, data received / used by those applications, and data output / generated by those applications.

[0019] The processor 120 and memory 122 collectively constitute processing means for controlling / automating the operation of the AVI station 110 and for processing images captured / generated by the AVI station 110 to detect the corresponding type of defect in the container and / or the contents of the container (e.g., a pharmaceutical sample). Specifically, as will be described in more detail below, for the AVI station 110-i, the processing means (120 and 122) are configured to (1) cause the imaging system 112 to capture images of the stopper edge of the container at appropriate times while the spinning means 118 spins the container, (2) generate a two-dimensional image of the stopper edge based on the set of images captured by the imaging system 112, and (3) process the pixels (e.g., pixel intensity values) of the obtained two-dimensional image using a trained neural network to generate output data. In alternative embodiments, the functions of the processor 120 and / or memory 122 are distributed across N different processing units and / or memory units, each specific to one of the AVI stations 110-1 through 110-N. In yet another embodiment, some of the functions of the processor 120 and memory 122 (e.g., functions for sample transport, spinning, and / or imaging) are distributed across the AVI stations 110, while other functions of the processor 120 and memory 122 (e.g., functions for generating 2D images from line scan camera images and / or functions for processing 2D images to detect defects) are performed at a centralized processing location. In some embodiments, at least a portion of the processor 120 and / or memory 122 is included in a computing system located outside (and possibly remotely) the line equipment 100.

[0020] Memory 122 stores the image 124 of the container / sample captured by AVI station 110, and also stores the AVI code 126. When the AVI code is executed by processor 120, it causes AVI station 110 to execute the corresponding functions as described above. For AVI station 110-i, for example, the AVI code 126 includes the corresponding part shown as code 128 in FIG. 1. As an example of one embodiment, code 128 may trigger the imaging system 112 to capture a line scan image while the sample is illuminated by the illumination system 114 and spun by the spin means 118, and may control the sample positioning hardware 116 to position the container in the correct position at the appropriate time. After the image is captured and stored in the image 124, code 128 processes each image 124 to detect defects associated with station 310-i (e.g., based on the number, size, and / or type of other objects such as particles and / or bubbles). As described above, in some embodiments, the part of code 128 that processes the image may be executed by a processor, component, and / or device different from the part of code 128 that controls conveyance, imaging, spinning, etc.

[0021] As shown in FIG. 1, the code 128 for the AVI station 110-i includes a sample movement and image capture unit 134 that generates commands / signals for controlling the conveying means 117 and the spinning means 118 as described above. The code 128 also includes an image generation unit 136, which constructs / generates different 2D images from line scan camera images for different containers. Further, the code 128 includes an inference model unit 138 that processes the 2D images generated by the image generation unit 136 using an inference model. The inference model includes a trained neural network (and may in some cases be entirely configured therefrom in the future), and the inference model processes pixels (e.g., intensity values, and in some cases color values) to generate output data indicating whether a particular sample is likely to be defective (e.g., there is a high likelihood of a number, size, and / or type of unacceptable particles on or near a stopper edge). The neural network and its training according to various exemplary embodiments are further described below with reference to FIGS. 5 and 6.

[0022] FIG. 2 shows in simplified form an existing (prior art) commercial line device 200, more specifically, the Bosch® AIM 5023 model. In one embodiment, the line device 200 is upgraded or modified using the techniques described herein. That is, after being so modified (e.g., by a field upgrade or a complete redesign of the product), the line device 200 can be used as the line device 100 of FIG. 1.

[0023] In production mode, the instrument 200 (Bosch® AIM 5023) is generally responsible for transporting, inspecting, and sorting syringes filled with solution (pharmaceuticals). The instrument 200 receives syringes from a De-Nester machine (e.g., Kyoto® G176 De-Nester) via a series of infeed screws and star wheels, after which automated inspection is initiated in the infeed (pre-inspection) unit and continued in the main unit. The infeed unit and main unit have various AVI stations, which are shown in Figure 2 as station 202 (several stations 202 are located in the same place, as indicated by two reference numbers in a single station). It should be understood that Figure 2 does not attempt to accurately or completely reconstruct the layout and components of the Bosch® AIM 5023. For example, various star wheels, discharge bins, and other components are not shown, and the relative positions shown for the various AVI stations 202 are not exactly correct.

[0024] In the infeed unit, the line equipment 200 includes the following three pre-inspection stations along a rotating star wheel 212A: (1) a bent needle shield inspection station 202-1 with charge-coupled device (CCD) cameras (referred to as "C01-1" inch and "C01-2" cameras); (2) a flange inspection station 202-2 with a CCD camera (referred to as "C02" camera); and (3) a stopper presence / color station 202-3 with a CCD camera (referred to as "C03" camera). These pre-inspections are based on a combination of technologies including CCD cameras, a stable light source, and an image processor. Syringes identified as defective at any of these stations 202-1 to 202-3 are discharged into the discharge area / bin (via star wheel 212A and another star wheel 212B) without being inverted or transferred to the main unit. However, the units that pass these inspections are inverted and transported to the main unit of equipment 200 via star wheel 212C.

[0025] In the main unit, the line equipment 200 includes 13 inspection stations along three rotary tables 210A-210C connected by two star wheels 212D and 212E. Specifically, along rotary table 210A, two inspection stations are located: (1) turbidity inspection station 202-4 equipped with a CCD camera (referred to as the "C04" camera); and (2) liquid color inspection station 202-5 equipped with a CCD camera (referred to as the "C05" camera). Five inspection stations are arranged along the rotating table 210B: (1) Body / fiber inspection station 202-6 having a CCD camera (referred to as "C1-1" and "C1-2" cameras); (2) Body (suspended particles) inspection station 202-7 having a CCD camera (referred to as "C2-1" and "C2-2" cameras); (3) Stopper edge inspection station 202-8 having a line scan CMOS camera (referred to as "C3-1" and "C3-2" cameras); (4) Stopper side inspection station 202-9 equipped with a CCD camera (referred to as "C4-1" and "C4-2" cameras); and (5) Stopper top inspection station 202-10 having a CCD camera (referred to as "C5-1" and "C5-2" cameras). A needle-shielded color inspection station 202-11, which has a CCD camera (referred to as the "C06" camera), is located on the star wheel 212E between the rotary tables 210B and 210C. Further along the rotary table 210C, five inspection stations are arranged: (1) particle inspection station 202-12 with CCD cameras (referred to as "C6-1" and "C6-2" cameras); (2) particle inspection station 202-13 using third-generation static segmentation (SDx) sensors (referred to as "SD1-1" and "SD1-2" sensors); (3) particle inspection station 202-14 with CCD cameras (referred to as "C7-1" and "C7-2" cameras); (4) particle inspection station 202-15 using SDx sensors (referred to as "SD2-1" and "SD2-2" sensors); and (5) fill level / air gap inspection station 202-16 with a CCD camera (referred to as "C8" camera).

[0026] Various stations 202-4 to 202-16 of the instrument 200 inspect the syringes as they are transported through the main unit. As part of the transport, the syringes are securely held by a freely rotating base attachment and spin cap. On the rotary table 210A, a spin motor is positioned in the peripheral area of ​​table 210A and uses a friction belt to spin the base attachment assembly, setting the appropriate spin for bubble dissipation and inspection. Rotary table 210B is equipped with an air knife ionizer that blows ionized air onto the syringe to remove any external particles or dust. On rotary tables 210B and 210C, the base mounting shaft relative to each syringe location is equipped with a direct spin function for proper inspection of visible particles in the solution. Each base attachment can be spun individually in a clockwise or counterclockwise direction at high or low speed.

[0027] After being processed through all inspection stations in the main unit, the syringes are discharged and sorted into either an "acceptance" route, where they are transported to another area and collected by a downstream machine (e.g., Kyoto® G176 Auto Trayer), or one of three discharge areas / stations. Each discharge station has a manually switchable discharge rail. Various rotary tables and / or star wheels may constitute means for transporting specific containers to designated rejection areas. For example, with respect to stations 202-8, star wheels 212E, 212F, 212G, and rotary table 210C, and optionally other star wheels, rails, and / or other mechanisms, may provide means for transporting containers / samples rejected at station 202-8 to appropriate rejection / discharge areas.

[0028] Returning to Figure 1, in one embodiment, line equipment 100 is modified to become equipment 200, and stopper edge inspection station 202-8 is modified to become AVI station 110-i (for example, using line scan cameras of imaging system 112, including one or both of the "C3-1" and "C3-2" cameras). In this embodiment, the transport means 117 includes a rotary table 210B (and optionally a unit for inverting each syringe), and the spin means 118 includes the free-rotating base attachment, spin cap, spin motor, and friction belt described above. In such an embodiment, specifically due to the improved accuracy of stopper edge inspection station 202-8, stopper top inspection station 202-10 may be omitted or modified (for example, to improve the detection accuracy of stations 202-10 and 202-8, depending on the context, by focusing on stopper defect detection rather than particle inspection).

[0029] Figures 3A and 3B show an exemplary container (syringe) 300, in which the stopper (plunger) 310 within a substantially cylindrical wall 312, particularly the edge of the plunger dome 314 (i.e., where the dome 314 meets the solution in the syringe 300), can be imaged using a line scan camera, such as the line scan camera of the imaging system 112. The wall 312 in which the plunger 310 is located may be made of translucent plastic, glass, or any other suitable material. In the particular orientation shown in Figures 3A and 3B (i.e., the plunger 310 is on the underside of the syringe 300), any large air pockets in the sample / solution in the syringe 300 should be well above the plunger dome 314 by the opposite (needle) end of the syringe 300.

[0030] As shown in the enlarged inset of Figure 3A, the line scan camera of the imaging system 112 is oriented so that for each rotational position of the syringe 300, the camera captures one vertical line image (sometimes referred to herein simply as “image”) corresponding to an area 322. Each line image captures only what is within a very narrow slice / area 322 at the time the image is captured. In Figure 3A, for example, the first line image may capture a portion of the object 330 (e.g., a particle or bubble), while the second line image (if the rotation is counterclockwise from the top) may capture another portion of the object 330. As the syringe 300 spins over 360 degrees (e.g., by the spinning means 118), the line scan camera captures enough line images (vertical slices / stacks of pixels) to cover the entire edge of the dome 314 of the plunger 310, as long as the images are captured in sufficiently small rotation increments (e.g., every 1 degree or every 3 degrees, depending on the image width of the line scan camera).

[0031] As shown in Figure 3B, the line scan camera may be angled slightly upward with respect to the horizontal plane (e.g., with respect to the plane of the flange of the syringe 300) to match or approximate the inclination of the plunger dome 314. In this way, particles, bubbles, or other objects located anywhere along the inclination of the dome 314 (e.g., near the apex, near the wall 312, or somewhere in between) are visible / shown as sharp relief against the relatively bright background provided by the illuminated solution in the syringe 300. Other orientations of the line scan camera relative to the syringe 300 are also possible.

[0032] Figure 4 shows an exemplary two-dimensional image 400 that may be generated from line images (e.g., vertical pixel stacks) captured by a line scan camera (for example, as the spinning means 118 rotates the syringe 300 of Figure 3 at least 360 degrees). Image 400 shows a stopper edge 402 (with a translucent solution on it), which may be generated by, for example, the image generation unit 136 of Figure 1. In the exemplary image 400, two objects 410, 412 (here, bubbles and glass particles, respectively) stationary on the stopper edge 402 can be seen relatively clearly from the profile image. For example, the stopper edge 402 may be the edge of a plunger dome 314, and object 410 or 412 may be object 330 in Figures 3A and 3B.

[0033] Figure 5 shows an exemplary neural network 500 that may be used to infer acceptability or unacceptability based on a two-dimensional image, such as the two-dimensional image 400 in Figure 4. The neural network 500 may be a trained neural network that forms (or is included in) an inference model, for example, implemented by the inference model unit 138 in Figure 1. The neural network 500 may be a convolutional neural network (CNN) or another preferred type of neural network. As shown in Figure 5, the exemplary neural network 500 includes an input layer 510, three hidden layers 512, and an output layer 514, each of which contains several nodes or “neurons”. In other embodiments, the neural network 500 may include more or fewer hidden layers 512 than three, and / or each layer may contain more or fewer nodes / neurons than shown in Figure 5.

[0034] The neural network 500 is trained to infer whether a particular two-dimensional image (e.g., image 400) is acceptable or unacceptable. It should be understood that "acceptable" may or may not mean that the corresponding sample does not require further inspection, and "unacceptable" may or may not mean that the corresponding sample must be discarded. In line equipment 100, for example, in order for a container / sample to pass quality inspection as a whole, the container / sample may need to successfully "pass" inspection at each of the AVI stations 110-1 through 110-N. In this case, an "acceptable" output at AVI station 110-i does not necessarily mean that the corresponding container / sample is usable (e.g., suitable for commercial sale or other use). As another example, in some embodiments, an "unacceptable" output at AVI station 110-i means that the container / sample must undergo additional (e.g., manual) inspection, not necessarily be rejected or discarded.

[0035] Referring to the line equipment 100 in Figure 1, the inference model unit 138 may pass values ​​(e.g., intensity values, and possibly RGB color values) of different pixels 502 of the image 400 to different neurons / nodes in the input layer 510. In some embodiments, the inference model unit 138 may preprocess the pixel values ​​(e.g., intensity and / or color values ​​from 0 to 255) before applying those values ​​to the input layer 510. As one simple example, the inference model unit 138 may convert each pixel value to a normalized value between 0 and 1. Other preprocessing is also possible (e.g., averaging multiple pixel values ​​within a pixel subset, or first cropping pixels from a relatively large area of ​​the image 400 so that the intensity values ​​do not change beyond a threshold amount and thus may represent the stopper body).

[0036] Figure 5 shows only four pixel values ​​passed to four neurons in the input layer 510, but in other embodiments, more pixel values ​​are passed to more neurons in the input layer 510, and as a result, the neural network 500 processes the image 400 in larger subsets or "chunks". In any case, in some embodiments, the inference model unit 138 may determine that the image 400 is "acceptable" only if the neural network 500 determines that all pixel subsets 502 are individually acceptable. In other, more complex embodiments, the neural network 500 may include three or more neurons in the output layer 514 to reflect the intermediate probabilities that non-bubble particles represent in a given pixel subset, and the inference model unit 138 jointly processes the results for all pixel subsets to determine, as a whole, whether the image 400 represents an acceptable sample or an unacceptable sample (specifically, at the stopper edge). In yet another embodiment, the neural network 500 has a large number of neurons in the input layer 510 to process all of the image 400 (or all pixels in the image 400 within a narrow horizontal band where the stopper meets the sample / solution) at once.

[0037] In some embodiments, each line connecting a first neuron to a second neuron in the neural network 500 is associated with a weight, the value of which is determined during the training process (further described below). The neural network 500 multiplies the value / output of the “source” neuron (i.e., the left side of the connection as seen in Figure 5) by its weight, and provides the multiplied value as input to a calculated function at the “destination” neuron (i.e., the right side of the connection as shown in Figure 5). Furthermore, each neuron in each hidden layer 512 may be associated with an “activation function” that acts on the input from the previous layer 510 or 512. For example, each neuron in each hidden layer 512 may have the following function applied:

number

number

number

number

number

[0038] Alternatively, a function other than the sigmoid function, such as the hyperbolic tangent (Tanh) function or the normalized linear unit (ReLU) function, may be applied to each neuron in the hidden layer 512.

[0039] It should be understood that many other embodiments are possible with respect to the arrangement of the neural network 500, the method by which pixel values ​​are preprocessed (e.g., averaging, segmenting, etc.) and / or provided to the neural network 500, and the method by which the output of the neural network 500 is processed or utilized by the inference model unit 138.

[0040] The neural network 500 may be trained using supervised learning. More specifically, the neural network 500 may be trained using a large set of 2D images (e.g., each similar to image 400) showing stopper edges at solution / stopper boundaries, along with a wide range of combined conditions. For example, the training images may include many different numbers, sizes, types, and locations of particles and / or bubbles, as well as different solution types (e.g., having different levels of translucency and possibly different viscosities) and / or other variations. Furthermore, each training image is labeled in such a way that it corresponds to a single correct or "true" output from the set of available outputs provided by the neural network 500 (e.g., "acceptable" or "unacceptable" in Figure 5). Labeling should be done carefully to ensure that all labels are correct (e.g., by manual inspection and possibly by laboratory testing). By using training samples under a sufficiently wide range of conditions, the neural network 500 can reliably distinguish between objects that were previously difficult to differentiate, such as gas-filled bubbles against heavy particles (e.g., glass particles).

[0041] Once the training dataset is complete, the neural network 500 can be trained. Any suitable training method may be used. For example, the neural network 500 may be trained for each training image using known techniques such as forward propagation, error calculation based on the inference result (e.g., mean squared error (MSE)), and backpropagation using gradient descent.

[0042] At a higher level, Figure 6 shows an exemplary development and certification process 600 for implementing deep learning using AVI stations such as station 110-i in Figure 1. During the development phase of process 600, labeled image data 602 is generated and / or collected for training purposes. The data 602 should be carefully selected and may include a number of 2D images showing stopper edges at the solution / stopper interface with a wide range of different conditions (e.g., particle size / type, bubbles, etc.), as described above. In stage 604, a machine learning algorithm trains a neural network (e.g., neural network 500 as discussed above) using the labeled image data.

[0043] Once the neural network is trained, in the certification phase of process 600, at stage 612, image data 610 (different from image data 602) is input to the trained model. The “trained model” may be the neural network alone, or it may include some additional modeling or processing (e.g., image data preprocessing before inputting the image data to the trained neural network). Throughout the certification process, the trained model is “locked.” That is, to ensure that the certification results remain valid, the model cannot be modified during or after the certification phase. This does not include, for example, improving the neural network with additional training data and thereby avoiding the risk of degrading the neural network’s performance (e.g., if the additional training images are improperly labeled).

[0044] In Stage 614, the results of the inference are observed for certification purposes. If the results show an acceptable level of accuracy (e.g., the rate of false positives and / or false negatives is sufficiently low for a sufficiently large sample size), the certification is successful and the model may be used for production. Whenever the model is modified (e.g., by further training / improving the model with images representing the new conditions), the certification phase should generally be repeated.

[0045] Figure 7 shows proof-of-concept results 700 and 720 obtained using neural network-based deep learning for a stopper edge inspection station (e.g., stopper edge inspection station 202-8 of Bosch® AIM 5023 line equipment). As can be seen in results 700 and 720, compared to operating the station without deep learning, deep learning increased detection capability by approximately 500% (5 times) and reduced false rejections by approximately 50% for this particular station.

[0046] Figure 8 is a flowchart of an exemplary method 800 for improving accuracy and efficiency in automated visual inspection of containers (e.g., syringes, vials, etc.). Method 800 may be implemented, for example, by the AVI station 110-i in Figure 1, in which the processor 120 executes the AVI code 128 in memory 122.

[0047] In method 800, in block 802, a container containing a sample (e.g., a liquid solution pharmaceutical) is oriented such that the line scan camera has a profile image of the edge of the container's stopper (e.g., a plunger or plug). For example, the container may be positioned relative to the line scan camera as shown in Figures 3A and 3B. Block 802 may be executed by the transport means 117 in Figure 1 in response to a command generated by the processor 120, for example, to cause the sample moving and image capture unit 134 to perform the operation.

[0048] In block 804, the container is spun by the spinning means 118 in response to a command generated by the processor 120, for example, to execute the sample movement and image capture unit 134. In block 806, while the container is spinning (e.g., during at least one full 360-degree rotation), multiple images of the stopper edge are captured using a line scan camera (e.g., the line scan camera of the imaging system 112). Each image is captured at various rotational positions of the container. It should be understood that, as used herein, an image is considered to have been captured "while the container was spinning" even if an image was captured when the container stopped. For example, in some embodiments, the timing of each image capture by the line scan camera may coincide with a short time when the container is stationary (e.g., stationary between discrete small rotational intervals while the container is spinning approximately 360 degrees step by step). Alternatively, the line scan camera can capture images at appropriate rotational positions of the container without requiring the container to stop spinning / rotating at any point during the line scan. Block 806 may be executed by the line scan camera of the imaging system 112 in response to a command generated by the processor 120, for example, to execute a sample movement and image capture unit 134.

[0049] In block 808, a two-dimensional image of the stopper edge is generated based on at least several images. Each image captured in block 806 may provide only one (or several) pixels on the first (e.g., horizontal) axis of the two-dimensional image, but may provide all pixels on the second (e.g., vertical) axis of the two-dimensional image. Block 808 may be executed, for example, by a processor 120 that causes an image generation unit 136 to run.

[0050] In block 810, pixels of a 2D image are processed by running an inference model, which includes a trained neural network (e.g., neural network 500 in Figure 5), to generate output data indicating the likelihood that a sample contains defects (e.g., based on the number, size, and / or type of particles or other objects in the sample that are at or near a stopper edge). In some embodiments, block 810 includes processing pixels of a 2D image by applying intensity values ​​associated with different pixels, or other values ​​derived from intensity values ​​(e.g., normalized values), to different nodes in the input layer of the trained neural network. Block 810 may be executed, for example, by a processor 120 running an inference model unit 138.

[0051] In some embodiments, method 800 includes one or more additional blocks not shown in Figure 8.

[0052] In one embodiment, for example, method 800 includes an additional block in which containers are selectively transported to a designated rejection area based on output data generated in block 810. This may be performed by additional transport means (e.g., additional rotary tables, star wheels, rails, etc., as discussed above with reference to Figure 2) in response to a command generated by processor 120 that causes sample movement and image capture unit 134 to perform the operation.

[0053] As another example, Method 800 may include blocks similar to blocks 802-806, but performed in parallel with blocks 802-806, for a second container / sample (i.e., to increase throughput). In such an embodiment, Method 800 may also include additional blocks, similar to blocks 808 and 810, in which additional two-dimensional images (of the stopper edge of the second container) are generated and processed.

[0054] While systems, methods, apparatus, and their components have been described in terms of exemplary embodiments, these systems, methods, apparatus, and their components are not limited to these. The detailed descriptions should be interpreted as illustrative examples only, and since it would be impractical, if not impossible, to describe all conceivable embodiments, not all possible embodiments of the present invention are described. Many alternative embodiments can be realized using either the current art or art developed after the filing date of this patent, but such embodiments still fall within the scope of the claims defining the present invention.

[0055] Those skilled in the art will understand that a wide variety of modifications, changes, and combinations can be made to the above embodiments without departing from the scope of the present invention, and that such modifications, changes, and combinations will be interpreted as being within the scope of the concept of the present invention.

Claims

1. 1. A method for increasing accuracy and efficiency in automated visual inspection of containers, comprising: orienting a container containing a sample such that a line scan camera obtains a profile image of an edge of a stopper of said container; spinning the vessel; capturing a plurality of images of the edge of the stopper with the line scan camera while spinning the container, each image of the plurality of images corresponding to a different rotational position of the container; generating, by one or more processors, a two-dimensional image of the edge of the stopper based on at least the plurality of images; processing pixels of the two-dimensional image with one or more processors executing an inference model including a trained neural network to generate output data indicative of the likelihood that the sample contains a defect; A method comprising:

2. The method of claim 1 , further comprising: causing, by one or more processors, the container to be selectively transported to a designated reject area based on the output data.

3. The method of claim 1 , wherein the output data indicates whether the sample contains one or more objects of a particular type.

4. The method of claim 3 , wherein the trained neural network is configured to distinguish between gas-filled bubbles and particles in the sample.

5. 2. The method of claim 1 , wherein processing the pixels of the two-dimensional image comprises applying intensity values ​​associated with different pixels, or other values ​​derived from the intensity values, to different nodes of an input layer of the trained neural network.

6. 2. The method of claim 1, wherein the container is a syringe, the stopper is a plunger, and the edge of the stopper is the edge of a plunger dome that contacts the sample.

7. The method of claim 1 , wherein orienting the container comprises transporting the container using a motorized rotary table or a star wheel.

8. The method of claim 1 , wherein orienting the container comprises inverting the container so that the stopper is below the sample.

9. The method of claim 1 , wherein spinning the container comprises rotating the container at least 360 degrees about a central axis of the container.

10. the line scan camera is a first line scan camera, the plurality of images is a first plurality of images, the container is a first container, and the two-dimensional image is a first two-dimensional image; and the method further comprises: While orienting the first container, orienting a second container so that a second line scan camera obtains a profile image of an edge of a stopper of the second container; spinning the first container while spinning the second container; capturing, with the second line scan camera, a second plurality of images of the edge of the stopper of the second container while the second container is spinning while capturing the first plurality of images, each image of the second plurality of images corresponding to a different rotational position of the second container; generating a second two-dimensional image based on at least the second plurality of images; The method of claim 1 further comprising:

11. 10. The method of claim 1, further comprising training the neural network using a labeled two-dimensional image of a stopper edge of a container prior to processing the pixels of the two-dimensional image.

12. 12. The method of claim 11, comprising training the neural network using labeled two-dimensional images of containers containing samples containing different types, numbers, sizes, and locations of objects.

13. A line scan camera, a conveying means for orienting a container containing a sample such that the line scan camera obtains a profile image of the edge of the container's stopper; a spinning means for spinning the container; A processing means, causing the line scan camera to capture a plurality of images of the edge of the stopper while the spinning means spins the container, each image of the plurality of images corresponding to a different rotational position of the container; generating a two-dimensional image of the edge of the stopper of the container based on at least the plurality of images; processing means for processing pixels of the two-dimensional image by executing an inference model comprising a trained neural network to generate output data indicative of whether the sample is acceptable; An automated visual inspection system comprising:

14. the conveying means is a first conveying means, and the automated visual inspection system is a second conveying means for conveying the container to a designated rejection area; The processing means further 14. The automated visual inspection system of claim 13, wherein the second transport means is configured to selectively transport the container to the designated reject area based on the output data.

15. 14. The automated visual inspection system of claim 13, wherein the output data indicates whether the sample contains one or more objects of a particular type.

16. 16. The automated visual inspection system of claim 15, wherein the trained neural network is configured to distinguish between gas-filled bubbles and particles in the sample.

17. 14. The automated visual inspection system of claim 13, wherein the processing means processes the pixels of the two-dimensional image by applying at least intensity values ​​associated with different pixels, or other values ​​derived from the intensity values, to different nodes of an input layer of the trained neural network.

18. 14. The automated visual inspection system of claim 13, wherein the container is a syringe, the stopper is a plunger, and the edge of the stopper is an edge of a plunger dome that contacts the sample.

19. 14. The automated visual inspection system of claim 13, wherein the transport means includes a motorized rotary table or a star wheel, and the transport means orients the container by transporting the container using at least the motorized rotary table or the star wheel.

20. 14. The automated visual inspection system of claim 13, wherein the transport means inverts the container so that the stopper is below the sample.

21. 14. The automated visual inspection system of claim 13, wherein said processing means causes said line scan camera to capture said plurality of images while said spinning means spins said container at least 360 degrees about a central axis of said container.

22. the line scan camera is a first line scan camera, the plurality of images are a first plurality of images, the container is a first container, the sample is a first sample, the conveying means is a first conveying means, the spinning means is a first spinning means, the two-dimensional image is a first two-dimensional image, and the output data is first output data; the automated visual inspection system further comprising a second line scan camera, a second transport means, and a second spin means; the second conveying means is for orienting a second container while the first conveying means is orienting the first container so that the second line scan camera obtains a profile image of an edge of a stopper of the second container; the second spin means for spinning the second container while the first spin means is spinning the first container; The processing means further causing the second line scan camera to capture a second plurality of images of the edge of the stopper of the second container while the first line scan camera is capturing the first plurality of images; generating a second two-dimensional image of the edge of the stopper of the second container based on at least the second plurality of images; and processing pixels of the second two-dimensional image by executing the inference model to generate second output data indicative of whether the second sample is acceptable.

23. 1. An automated visual inspection system comprising: A line scan camera, sample positioning hardware configured to orient a vessel containing a sample such that the line scan camera obtains a profile image of an edge of a stopper of the vessel, and to spin the vessel while so oriented; A memory storing instructions that, when executed by one or more processors, cause the one or more processors to: causing the line scan camera to capture a plurality of images of the edge of the stopper while the container is spinning, each image of the plurality of images corresponding to a different rotational position of the container; generating a two-dimensional image of the edge of the stopper of the container based on at least the plurality of images; a memory for executing an inference model including a trained neural network to process pixels of the two-dimensional image and generate output data indicating whether the sample is acceptable; and An automated visual inspection system comprising:

24. 24. The automated visual inspection system of claim 23, wherein the output data indicates whether the sample contains one or more objects of a particular type.

25. 25. The automated visual inspection system of claim 24, wherein the trained neural network is configured to distinguish between gas-filled bubbles and particles in the sample.

26. 24. The automated visual inspection system of claim 23, wherein the instructions cause the one or more processors to process the pixels of the two-dimensional image by applying at least intensity values ​​associated with different pixels or other values ​​derived from the intensity values ​​to different nodes of an input layer of the trained neural network.

27. 24. The automated visual inspection system of claim 23, wherein the container is a syringe, the stopper is a plunger, and the edge of the stopper is the edge of a plunger dome that contacts the sample.

28. 24. The automated visual inspection system of claim 23, wherein the sample positioning hardware includes a motorized rotary table or star wheels, and at least the motorized rotary table or star wheels are used to transport the container and thereby orient the container.

29. 24. The automated visual inspection system of claim 23, wherein the sample positioning hardware inverts the container so that the stopper is below the sample.

30. The instructions cause the one or more processors to:

24. The automated visual inspection system of claim 23, wherein the linescan camera captures the plurality of images while the container is spun at least 360 degrees about a central axis of the container.

Images