Targeted application of deep learning to automated visual inspection equipment

Abstract

To provide a method for enhancing accuracy and efficiency in automated visual inspection of a vessel (for example, syringe, vial, etc.), and an automated visual inspection system.SOLUTION: In a method, a vessel containing a sample is oriented such that a line scan camera acquires a profile image of an edge of a stopper of the vessel. A plurality of images of the edge of the stopper is captured by a first line scan camera while spinning the vessel. Each image of the plurality of images corresponds to a different rotational position of the vessel. A two-dimensional image of the edge of the stopper is generated based on at least the plurality of images, and pixels of the two-dimensional image are processed, by one or more processors executing an inference model that includes a trained neural network, to generate output data indicative of a likelihood that the sample is defective.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] This application generally relates to an automated visual inspection (AVI) system for pharmaceuticals or other products, and more specifically to techniques for detecting and differentiating particles and other objects (such as air bubbles) within containers filled with a sample (e.g., a solution).

Background Art

[0002] In certain situations, such as in quality control procedures for manufactured pharmaceuticals, it is necessary to examine whether a sample (e.g., a container / vessel such as a syringe or vial, and / or the contents such as a liquid or lyophilized pharmaceutical) is free of defects. The acceptability of a particular sample under applicable quality criteria may depend on measurement criteria such as, for example, the type and / or size of container defects (e.g., chips or cracks), or the type, number, and / or size of undesirable particles (e.g., fibers) in the pharmaceutical. If a sample has non - acceptable measurement criteria, it may be rejected and / or discarded.

[0003] To handle the volumes typically associated with commercial production of pharmaceuticals, defect inspection tasks are increasingly being automated. However, the automated detection of particulate matter in solutions presents particular challenges in the pharmaceutical industry. Achieving high detection accuracy is generally difficult and is made even more so by higher viscosity solutions that impede the movement of particles that might otherwise indicate particle types. In the case of protein - based products with formulations that release gases promoting bubble formation, conventional particle detection techniques can result in particularly high rates of false rejections. For example, with such techniques, it can be difficult to distinguish these air bubbles (which may adhere to the container) from heavy particles that tend to sediment / settle against a portion of the container (e.g., against the plunger of a syringe filled with solution).

[0004] Moreover, dedicated equipment used to assist in automatic defect inspection has become very large, very complex, and very expensive. A single commercial line device may include a number of different AVI stations, and each of the AVI stations may handle different specific inspection tasks. As just one example, the Bosch® Automatic Inspection Machine (AIM) 5023 commercial line device used in the filling completion inspection stage of a drug-filled syringe includes 14 separate visual inspection stations with 16 general inspection tasks and a number of cameras and other sensors. Overall, such equipment may be designed to detect a wide range of defects, including defects in the integrity of the container such as large cracks or container closure, cosmetic container defects such as scratches or dirt on the container surface, and defects related to the pharmaceutical product itself such as the color of the liquid or the presence of foreign particles. However, due to the above-described problems associated with particle detection and characterization, such equipment may require redundancy between AVI stations. For example, in the case of the Bosch® AIM 5023 line device, due to the relatively low performance of the "stopper edge" inspection station (for detecting and differentiating heavy particles stationary on the dome of the syringe plunger), it may also be necessary to perform particle inspection at another "stopper top" AVI station with additional cameras to bring the overall level of particle inspection accuracy to an acceptable level. This increases the complexity and cost of the equipment and / or requires adapting the "stopper top" AVI station to perform multiple inspection tasks rather than being optimized for a single task (e.g., detecting defects in the stopper itself). Summary of the Invention Means for Solving the Problems

[0005] The embodiments described herein relate to systems and methods in which deep learning is applied to a particular type of AVI station (e.g., within commercial in-line equipment that may include multiple AVI stations) to significantly improve accuracy by synergistic action (e.g., with far fewer false rejections and / or false positives). Additionally or alternatively, the described systems and methods may enable advantageous changes to other AVI stations (e.g., within the same commercial in-line equipment) by allowing other AVI stations to focus on other tasks and / or by completely eliminating other AVI stations.

[0006] Specifically, deep learning is applied to an AVI station that utilizes one or more line scan cameras (e.g., CMOS line scan cameras), and the line scan cameras detect and distinguish objects that are stationary or placed at or near the edge of a stopper of a container containing a sample (e.g., a liquid solution pharmaceutical) (e.g., glass and / or other particles relative to gas-filled bubbles). For example, the AVI station may utilize a line scan camera to detect and distinguish an object that is placed on or near the surface of a syringe plunger dome that is in contact with a liquid sample within the syringe. The line scan camera may capture multiple line images as the AVI station rotates / spins the container at least one full rotation (360 degrees), and then a processing device or a component within the AVI station (or communicatively coupled to the AVI station) generates a two-dimensional image from the multiple line images.

[0007] An AVI station or an external processing component provides pixel values of a two-dimensional image (e.g., normalized pixel intensity values) to a trained neural network. Thereby, it is inferred whether the container sample is unacceptable (e.g., whether the particles in the imaged area contain an unacceptable number, size, and / or type). The neural network may be trained using a supervised learning method with a wide range of two-dimensional images of samples that are known (and labeled) to have acceptable or unacceptable numbers, types, sizes, etc. of particles and / or gas-filled bubbles. The selection and classification of the images used to train the neural network are important for performance in the inference phase. Further, to avoid accepting defective units, unexpected situations should be predicted and included in the training images. Importantly, the trained neural network, or a larger inference model including the neural network, may be "locked" prior to certification so that the model cannot be changed (e.g., further trained) without recertification. To ensure that the system functions as well as or better than manual visual inspection, preferably, acceptance criteria should be established and pre-approved.

[0008] If the AVI station (or a communicatively coupled processing device) indicates that the sample has a defect, the AVI station, or the commercial line equipment including the AVI station, physically transports the container / sample to a rejection area where the sample can be discarded / destroyed or transferred for further inspection (e.g., manual inspection). Depending on the embodiment, the container / sample may be transported directly to a discharge / rejection area (e.g., a bin) or may first pass through one or more other AVI stations. If the inference model does not indicate that the sample has a defect, the AVI station or the commercial line equipment may directly transport the container / sample to an area designated for accepted products or to the next AVI station (e.g., one or more AVI stations designed to detect defects in other types of samples and / or containers) for further inspection.

[0009] It will be understood by those skilled in the art that the figures described in this specification are included for illustrative purposes and do not limit the present disclosure. The drawings are not necessarily to scale and instead focus on showing the principles of the present disclosure. In some cases, it should be understood that various aspects of the described embodiments may be shown exaggerated or enlarged to facilitate understanding of the described embodiments. In the drawings, like reference numerals throughout the various drawings generally refer to components that are functionally and / or structurally similar.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Best Mode for Carrying Out the Invention

[0011] The various concepts introduced above and described in more detail below can be realized in any of a number of ways, and the concepts described are not limited to any particular implementation. Examples of implementations are provided for illustrative purposes.

[0012] FIG. 1 is a simplified block diagram of an exemplary line device 100 in which the technology described herein may be implemented. The line device 100 can be any production-grade device having, for example, N (N≧1) AVI stations 110-1 to 110-N (collectively also referred to as AVI stations 110). To provide a more specific example, the line device 100 may be a modified version of a Bosch (registered trademark) Automatic Inspection Machine (AIM) 5023 commercial line device, which is discussed below with reference to FIG. 2. Each of the AVI stations 110 may be responsible for capturing images used for inspecting various aspects of a container (e.g., syringe, vial, etc.) and / or a sample (e.g., liquid solution pharmaceutical) within the container. For example, the first AVI station 110-1 may capture an image of the top view of a syringe, vial, or other container to inspect for cracks or chips, and the second AVI station 110-2 (not shown in FIG. 1) may capture a side image to inspect for the presence or absence of foreign particles throughout the sample within the container.

[0013] FIG. 1 shows in the form of a simplified block diagram the general components of the i-th AVI station 110-i, where i may be any integer from 1 to N. The AVI station 110-i is specifically configured to automatically visually inspect a sample (the contents of the container) in the area where the sample meets / comes into contact with the edge of the stopper of the container. The stopper may be, for example, the plunger of a syringe, or a cap or plug that seals the opening of a vial. To perform this inspection, the AVI station 110-i includes an imaging system 112, an illumination system 114, and sample positioning hardware 116. (If any) Other AVI stations 110 may have generally similar types of components (e.g., imaging systems, illumination systems, and sample positioning hardware), but it should be understood that in some cases, they may have different types and configurations of components appropriate for each given station 110.

[0014] The imaging system 112 includes at least one line scan camera and, optionally, associated optical components (e.g., additional lenses, mirrors, filters, etc.) to capture a line image of each sample (pharmaceutical). Each of the line scan cameras may be, for example, a CMOS line scan camera. For the sake of simplicity of explanation, most of the following description refers only to a single line scan camera. However, it should be understood that multiple line scan cameras may be used. For example, to improve throughput, each of two line scan cameras may image different containers / samples simultaneously in parallel.

[0015] The illumination system 114 includes one or more illumination devices to illuminate each sample while the sample is being imaged by the line scan camera. The illumination device may include one or more light emitting diodes (LEDs), such as an LED array arranged as a backlight panel.

[0016] The sample positioning hardware 116 may include any hardware that holds (or otherwise supports) and moves a container for the AVI station 110-i. In the embodiment of FIG. 1, the sample positioning hardware 116 includes at least a conveying means 117 for orienting each container so that the line scan camera of the imaging system 112 obtains a profile image of the edge of the stopper of the container, and a spinning means 118 for spinning each container (e.g., rotating about the central axis of the container) while the line scan camera is capturing a line image. The conveying means 117 may include an electric rotary table, a star wheel or carousel, a robotic arm, and / or any other suitable mechanism for orienting (e.g., moving and positioning) each container. The spinning means 118 may include, for example, an electric spinning mechanism (e.g., a component of Bosch® AIM 5023 that provides a “direct spin” function for syringes, as discussed below with reference to FIG. 2). As further explained below, after the conveying means 117 properly positions / orients a given container, the spinning means 118 spins the container so that the line scan camera can capture a line image that comprehensively covers a 360-degree drawing of the stopper in the area where the stopper contacts the sample.

[0017] In some embodiments, the sample positioning hardware 116 also includes hardware for inverting each container (e.g., to ensure that the stopper is positioned under the sample when imaging is performed, increasing the likelihood that heavy particles are resting directly on the stopper) and / or for agitating the sample contained within each container. In other embodiments, the particular manner of properly orienting each container (e.g., container inversion) occurs at the first AVI station 110, within the first AVI station 110, or prior to handling by line equipment 100 or the like. Various exemplary orientations of the line scan camera with respect to the container / sample when the line scan camera captures an image of the spinning sample are described below with reference to FIGS. 3A and 3B.

[0018] The line device 100 also includes one or more processors 120 and a memory 122. Each of the processors 120 may be a programmable microprocessor that executes software instructions stored in the memory 122 to perform some or all of the software control functions of the line device 100 as described herein. Alternatively or in addition, one or more of the processors 120 may be other types of processors (e.g., application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc.), and some of the functions of the processors 120 described herein may be implemented in hardware instead. The memory 122 may include one or more volatile and / or non-volatile memories. Any suitable type of memory, such as read only memory (ROM), random access memory (RAM), flash memory, solid state drive (SSD), hard disk drive (HDD), etc., may be included in the memory 122. The memory 122 may collectively store one or more software applications, the data received / used by those applications, and the data output / generated by those applications.

[0019] Processor 120 and memory 122 collectively constitute processing means for controlling / automating the operation of AVI station 110 and for processing images captured / generated by AVI station 110 to detect corresponding types of defects with respect to the container and / or the contents of the container (e.g., pharmaceutical samples). Specifically, as will be described in more detail below, for AVI station 110-i, the processing means (120 and 122) are configured to: (1) cause imaging system 112 to capture an image of the stopper edge of the container at an appropriate time while spin means 118 is spinning the container; (2) generate a two-dimensional image of the stopper edge based on a set of images captured by imaging system 112; and (3) process the pixels (e.g., pixel intensity values) of the resulting two-dimensional image using a trained neural network to generate output data. In an alternative embodiment, the functions of processor 120 and / or memory 122 are distributed among N different processing units and / or memory units, each of which is unique to a different one of AVI stations 110-1 to 110-N. In yet another embodiment, some of the functions of processor 120 and memory 122 (e.g., functions for sample conveyance, spinning, and / or imaging) are distributed across AVI station 110, while other functions of processor 120 and memory 122 (e.g., functions for generating a two-dimensional image from line scan camera images and / or for processing the two-dimensional image to detect defects) are executed at a centralized processing location. In some embodiments, at least a portion of processor 120 and / or memory 122 is included in a computing system external to (and possibly remote from) line equipment 100.

[0020] The memory 122 stores the images 124 of the containers / samples captured by the AVI station 110, and also stores the AVI code 126. When the AVI code is executed by the processor 120, it causes the AVI station 110 to execute the corresponding functions as described above. For the AVI station 110-i, for example, the AVI code 126 includes the corresponding part shown as code 128 in FIG. 1. As an example of an embodiment, the code 128 may trigger the imaging system 112 to capture line scan images while the sample is illuminated by the illumination system 114 and spun by the spin means 118, and may control the sample positioning hardware 116 to position the container at the correct position at the appropriate time. After the image is captured and stored in the image 124, the code 128 processes each image 124 to detect defects associated with the station 310-i (e.g., based on the number, size, and / or type of other objects such as particles and / or bubbles). As described above, in some embodiments, the part of the code 128 that processes the images may be executed by a processor, component, and / or device that is different from the part of the code 128 that controls conveyance, imaging, spinning, etc.

[0021] As shown in FIG. 1, the code 128 for the AVI station 110-i includes a sample movement and image capture unit 134 that generates commands / signals for controlling the conveying means 117 and the spinning means 118 as described above. The code 128 also includes an image generation unit 136 that constructs / generates different two-dimensional images from line scan camera images for different containers. Further, the code 128 includes an inference model unit 138 that processes the two-dimensional images generated by the image generation unit 136 using an inference model. The inference model includes a trained neural network (and may in some cases be fully configured from scratch), and the inference model processes pixels (e.g., intensity values, and in some cases color values) to generate output data indicating whether a particular sample is likely to be defective (e.g., there is a high likelihood of an unacceptable number, size, and / or type of particles on or near the stopper edge). The neural network and its training according to various exemplary embodiments are further described below with reference to FIGS. 5 and 6.

[0022] FIG. 2 shows in a simplified form an existing (prior art) commercial line device 200, more specifically, the Bosch® AIM 5023 model. In one embodiment, the line device 200 is upgraded or modified using the techniques described herein. That is, after being so modified (e.g., by a field upgrade or a complete redesign of the product), the line device 200 can be used as the line device 100 of FIG. 1.

[0023] In the production mode, the device 200 (Bosch (registered trademark) AIM 5023) generally undertakes the roles of transporting, inspecting, and sorting syringes filled with a solution (pharmaceutical). The device 200 receives syringes from a denester machine (e.g., Kyoto (registered trademark) G176 De-Nester) via a series of infeed screws and star wheels, and then an automatic inspection is initiated in the infeed (pre-inspection) unit and continued in the main unit. The infeed unit and the main unit have various AVI stations, which are shown in Figure 2 as station 202 (several stations 202 are arranged in the same location as indicated by two reference numbers in a single station). It should be understood that Figure 2 does not attempt to accurately or completely reconstruct the layout and components of Bosch (registered trademark) AIM 5023. For example, various star wheels, discharge bins, and other components are not shown, and the relative positions shown for the various AVI stations 202 are not exactly correct.

[0024] In the infeed unit, the in-line device 200 includes the following three pre-inspection stations along the rotating star wheel 212A. (1) A vent needle shield inspection station 202-1 with charge-coupled device (CCD) cameras (referred to as "C01-1" and "C01-2" cameras); (2) A flange inspection station 202-2 equipped with a CCD camera (referred to as the "C02" camera); and (3) A stopper presence / color station 202-3 equipped with a CCD camera (referred to as the "C03" camera). These pre-inspections are based on a combination of technologies including CCD cameras, stable light sources, and image processors. In any of these stations 202-1 to 202-3, syringes identified as having defects are discharged into the discharge area / bin (via star wheel 212A and another star wheel 212B) without being inverted or transferred to the main unit. However, units that pass these inspections are inverted and transported to the main unit of the device 200 via star wheel 212C.

[0025] In the main unit, the line equipment 200 includes 13 inspection stations along three rotating tables 210A to 210C coupled by two star wheels 212D and 212E. Specifically, along the rotating table 210A, two inspection stations are arranged: (1) a turbidity inspection station 202-4 equipped with a CCD camera (referred to as the "C04" camera); and (2) a liquid color inspection station 202-5 equipped with a CCD camera (referred to as the "C05" camera). Along the rotating table 210B, five inspection stations are arranged: (1) a body / fiber inspection station 202-6 having CCD cameras (referred to as the "C1-1" and "C1-2" cameras); (2) a body (floating particle) inspection station 202-7 having CCD cameras (referred to as the "C2-1" and "C2-2" cameras); (3) a stopper edge inspection station 202-8 having line scan CMOS cameras (referred to as the "C3-1" and "C3-2" cameras); (4) a stopper side inspection station 202-9 equipped with CCD cameras (referred to as the "C4-1" and "C4-2" cameras); (5) a stopper upper inspection station 202-10 having CCD cameras (referred to as the "C5-1" and "C5-2" cameras). There is a needle shield color inspection station 202-11 having a CCD camera (referred to as the "C06" camera) on the star wheel 212E between the rotating tables 210B and 210C. Along the rotating table 210C, five more inspection stations are arranged: (1) a particle inspection station 202-12 having CCD cameras (referred to as the "C6-1" and "C6-2" cameras); (2) a particle inspection station 202-13 using third-generation static division (SDx) sensors (referred to as the "SD1-1" and "SD1-2" sensors); (3) a particle inspection station 202-14 having CCD cameras (referred to as the "C7-1" and "C7-2" cameras); (4) a particle inspection station 202-15 using SDx sensors (referred to as the "SD2-1" and "SD2-2" sensors); (5) a filling level / air gap inspection station 202-16 having a CCD camera (referred to as the "C8" camera).

[0026] The various stations 202-4 to 202-16 of the machine 200 inspect the syringes as they are transported through the main unit. As part of the transportation, the syringes are firmly held by a freely rotating base attachment and a spin cap. On the rotary table 210A, a spin motor is arranged in the peripheral area of the table 210A, and a friction belt is used to spin the base attachment assembly to set an appropriate spin for the dissipation of air bubbles and inspection. The rotary table 210B is equipped with an air knife ionizer that blows ionized air onto the syringe to remove any external particles or dust. On the rotary tables 210B and 210C, the base mounting shafts for each syringe location are equipped with a direct spin function to appropriately inspect visible particles in the solution. Each base attachment can be individually spun at high or low speed, in a clockwise or counterclockwise direction.

[0027] After being processed through all the inspection stations of the main unit, the syringes are discharged and transported to another area and classified into either an "acceptance" route where they are collected by a downstream machine (e.g., Kyoto (registered trademark) G176 Auto Trayer, etc.), or one of three discharge areas / stations. Each discharge station has a manually switchable release discharge rail. Various rotary tables and / or star wheels may constitute means for transporting a specific container to a designated rejection area. For example, with respect to station 202-8, the star wheels 212E, 212F, 212G, and the rotary table 210C, and optionally other star wheels, rails, and / or other mechanisms, may provide means for transporting the containers / samples rejected at station 202-8 to an appropriate rejection / discharge area.

[0028] Returning to FIG. 1, in one embodiment, the line equipment 100 is changed to equipment 200, and the stopper edge inspection station 202-8 is changed to the AVI station 110-i (e.g., using the line scan camera of the imaging system 112, including one or both of the "C3-1" and "C3-2" cameras). Also, in this embodiment, the conveying means 117 includes a rotary table 210B (and optionally, a unit for inverting each syringe), and the spinning means 118 includes the freely rotatable base attachment, spin cap, spin motor, and friction belt described above. In such an embodiment, specifically due to the improved accuracy of the stopper edge inspection station 202-8, the stopper upper inspection station 202-10 may be omitted or changed (e.g., by focusing on stopper defect detection rather than particle inspection, optionally to improve the detection accuracy of both station 202-10 and station 202-8).

[0029] FIGS. 3A and 3B show an exemplary container (syringe) 300, where the edge of the stopper (plunger) 310, particularly the edge of the plunger dome 314, within the generally cylindrical wall 312 (i.e., the location where the dome 314 meets the solution in the syringe 300), can be imaged using a line scan camera such as the line scan camera of the imaging system 112. The wall 312 in which the plunger 310 is disposed may be made of translucent plastic, glass, or any other suitable material. In the particular orientation shown in FIGS. 3A and 3B (i.e., the plunger 310 is on the lower side of the syringe 300), any large air pockets in the sample / solution within the syringe 300 should be well above the plunger dome 314 by the opposite (needle) end of the syringe 300.

[0030] As shown in the enlarged insert of FIG. 3A, the line scan camera of the imaging system 112 is oriented such that for each rotational position of the syringe 300, the camera captures one vertical line image (which may also be simply referred to as an “image” herein) corresponding to the area 322. Each line image captures only what is within a very narrow slice / area 322 at the time the image is captured. In FIG. 3A, for example, the first line image may capture a portion (e.g., a particle or a bubble) of the object 330, while the second line image (when the rotation is counterclockwise from the top surface) may capture another portion of the object 330. As the syringe 300 spins through 360 degrees (e.g., by the spin means 118), the line scan camera captures a sufficient number of line images (vertical slices / stacks of pixels) to cover the entire edge of the dome 314 of the plunger 310, as long as the images are captured in small enough rotational increments (e.g., once per degree, or once every three degrees, depending on the image width of the line scan camera).

[0031] As shown in FIG. 3B, the line scan camera may be angled slightly upward with respect to the horizontal plane (e.g., with respect to the plane of the flange of the syringe 300) so as to align with or approximate the slope of the plunger dome 314. In this way, particles, bubbles, or other objects anywhere along the slope of the dome 314 (e.g., near the apex, near the wall 312, or somewhere in between) appear / show as distinct relief against the relatively bright background provided by the illuminated solution in the syringe 300. Other orientations of the line scan camera with respect to the syringe 300 are possible.

[0032] FIG. 4 shows an exemplary two-dimensional image 400 that can be generated from a line image (e.g., a vertical pixel stack) captured by a line scan camera as (for example) the spin means 118 rotates the syringe 300 of FIG. 3 by at least 360 degrees. Image 400 shows a stopper edge 402 (on which there is a translucent solution) and may be generated, for example, by the image generation unit 136 of FIG. 1. In the exemplary image 400, two objects 410, 412 (here a bubble and a glass particle, respectively) stationary on the stopper edge 402 can be seen relatively clearly from the profile image. For example, the stopper edge 402 may be the edge of the plunger dome 314, and the object 410 or 412 may be the object 330 of FIGS. 3A and 3B.

[0033] FIG. 5 shows an exemplary neural network 500 that can be used to infer acceptability or non-acceptability based on a two-dimensional image such as the two-dimensional image 400 of FIG. 4. The neural network 500 may be a trained neural network that forms (or is included in) an inference model implemented, for example, by the inference model unit 138 of FIG. 1. The neural network 500 may be a convolutional neural network (CNN) or another suitable type of neural network. As shown in FIG. 5, the exemplary neural network 500 includes an input layer 510, three hidden layers 512, and an output layer 514, each of which includes several nodes or "neurons". In other embodiments, it should be understood that the neural network 500 may include more or fewer than three hidden layers 512 and / or each layer may include more or fewer nodes / neurons than shown in FIG. 5.

[0034] The neural network 500 is trained to infer whether a particular two-dimensional image (e.g., image 400) is acceptable or unacceptable. It should be understood that "acceptable" may or may not mean that the corresponding sample does not require further inspection, and "unacceptable" may or may not mean that the corresponding sample must be discarded. In the line equipment 100, for example, in order for a container / sample to pass the quality inspection as a whole, the container / sample may need to successfully "pass" the inspections at each of the AVI stations 110-1 to 110-N. In this case, an "acceptable" output at the AVI station 110-i does not necessarily mean that the corresponding container / sample is usable (e.g., suitable for commercial sale or other uses). As another example, in some embodiments, an "unacceptable" output at the AVI station 110-i means that the container / sample is not necessarily rejected or discarded, but rather must undergo additional (e.g., manual) inspection.

[0035] Referring to the line equipment 100 of FIG. 1, the inference model unit 138 may pass the values of different pixels 502 of the image 400 (e.g., intensity values and, optionally, RGB color values) to different neurons / nodes of the input layer 510. In some embodiments, the inference model unit 138 may preprocess the pixel values (e.g., intensity and / or color values between 0 and 255, etc.) before applying them to the input layer 510. As a simple example, the inference model unit 138 may convert each pixel value to a normalized value between 0 and 1. Other preprocessing (e.g., averaging multiple pixel values within a pixel subset, or first cropping pixels for a relatively large area of the image 400 so that the intensity values do not change beyond a threshold amount and thus may potentially represent a stopper body) is also possible.

[0036] FIG. 5 shows only the four pixel values passed to the four neurons of the input layer 510, but in other embodiments, more pixel values are passed to more neurons of the input layer 510, and as a result, the neural network 500 processes the image 400 in larger subsets or "chunks". In any case, in some embodiments, the inference model unit 138 may determine that the image 400 is "acceptable" only if the neural network 500 determines that all pixel subsets 502 are individually acceptable. In other, more complex embodiments, the neural network 500 may include three or more neurons in the output layer 514 to reflect an intermediate probability that non-bubble particles are shown in a given pixel subset, and the inference model unit 138 jointly processes the results for all pixel subsets to determine, as a whole, whether the image 400 represents an acceptable sample or an unacceptable sample (specifically, at the stopper edge). In still other embodiments, the neural network 500 has a large number of neurons in the input layer 510 and processes all of the image 400 (or all of the pixels within the narrow horizontal band in the image 400 where the stopper meets the sample / solution) at once.

[0037] In some embodiments, each line connecting a first neuron to a second neuron in the neural network 500 is associated with a weight, the value of which is determined during the training process (described further below). The neural network 500 multiplies the weight by the value / output of the "source" neuron (i.e., the left side of the connection as seen in FIG. 5) and provides the multiplied value as an input to a function calculated at the "destination" neuron (i.e., the right side of the connection as shown in FIG. 5). Further, each neuron in each hidden layer 512 may be associated with an "activation function" that acts on the input from the previous layer 510 or 512. For example, the neurons in each hidden layer 512 may apply the following function.

Equation

[0038] Instead, functions other than the sigmoid function, such as the hyperbolic tangent (Tanh) function or the rectified linear unit (ReLU) function, for example, may be applied to each neuron in the hidden layer 512.

[0039] It should be understood that many other embodiments are possible with respect to how the pixel values are preprocessed (e.g., averaged, segmented, etc.) and / or provided to the neural network 500, and with respect to how the output of the neural network 500 is processed or utilized by the inference model unit 138.

[0040] The neural network 500 may be trained using supervised learning. More specifically, the neural network 500 may be trained using a large set of two-dimensional images showing the stopper edge at the solution / stopper boundary (e.g., each similar to image 400), along with a wide variety of combined conditions. For example, the training images may include many different numbers, sizes, types, and positions of particles and / or bubbles, and in some cases different solution types (e.g., having different levels of translucency and in some cases different viscosities), and / or other variations. Further, each training image is labeled in a form corresponding to a single correct or "true" output from the set of available outputs provided by the neural network 500 (e.g., in FIG. 5, "acceptable" or "not acceptable"). To ensure that all labels are correct, the labeling should be done carefully (e.g., by manual inspection and in some cases by laboratory tests). By using the training samples under a sufficiently wide range of conditions, the neural network 500 can reliably distinguish objects that were previously difficult to distinguish, such as gas-filled bubbles from heavy particles (e.g., glass particles).

[0041] Once the training dataset is complete, the neural network 500 can be trained. Any suitable training technique may be used. For example, the neural network 500 may be trained using known techniques of forward propagation, error calculation based on the inference result (e.g., mean squared error (MSE)), and backpropagation using gradient descent for each training image.

[0042] At a higher level, FIG. 6 shows an exemplary development and certification process 600 for implementing deep learning using an AVI station such as station 110-i of FIG. 1. In the development stage of process 600, for training purposes, labeled image data 602 is generated and / or collected. The data 602 should be carefully selected and may include a number of two-dimensional images showing stopper edges at the solution / stopper interface with a wide variety of conditions as described above (e.g., particle size / type, bubbles, etc.). At stage 604, a machine learning algorithm uses the labeled image data to train a neural network (e.g., neural network 500 as discussed above).

[0043] Once the neural network is trained, in the certification stage of process 600, at stage 612, image data 610 (different from image data 602) is input into the trained model. The "trained model" may be the neural network alone or may include some additional modeling or processing (e.g., preprocessing of the image data before inputting the image data into the trained neural network). Throughout the certification, the trained model is "locked". That is, the model cannot be modified during or after the certification phase to ensure that the certification results remain valid. This includes, for example, not including the risk of improving the neural network with additional training data and thereby degrading the performance of the neural network (e.g., if the additional training images are inappropriately labeled).

[0044] At stage 614, the results of the inference are observed for certification purposes. If the results show an acceptable level of accuracy (e.g., the rate of false positives and / or false negatives is sufficiently low for a sufficiently large sample size), the certification is successful and the model may be used in production. Whenever the model is changed (e.g., by further training / improving the model using images representing new conditions), generally the certification phase must be repeated.

[0045] Figure 7 shows proof-of-concept results 700, 720 obtained using neural network-based deep learning for a stopper edge inspection station (e.g., similar to the stopper edge inspection station 202-8 of Bosch® AIM 5023 line equipment). As can be seen from results 700 and 720, for this particular station, deep learning increased the detection ability by approximately 500% (5 times) and decreased false rejections by approximately 50% compared to operating the station without deep learning.

[0046] Figure 8 is a flowchart of an exemplary method 800 for enhancing accuracy and efficiency in the automatic visual inspection of containers (e.g., syringes, vials, etc.). Method 800 may be implemented, for example, by the AVI station 110-i of FIG. 1, and the processor 120 executes the AVI code 128 within the memory 122.

[0047] In method 800, at block 802, a container containing a sample (e.g., a liquid solution pharmaceutical) is oriented such that a line scan camera has a profile image of the edge of a stopper (e.g., a plunger or plug) of the container. For example, the container may be positioned relative to the line scan camera as shown in FIGS. 3A and 3B. Block 802 may be performed by the conveyance means 117 of FIG. 1 in response to a command generated by the processor 120 that causes, for example, the sample movement and image capture unit 134 to execute.

[0048] In block 804, the container is spun by spin means 118 in response to a command generated by processor 120 that causes, for example, sample movement and image capture unit 134 to execute. In block 806, while the container is spinning (e.g., during at least one complete 360-degree rotation), a plurality of images of the stopper edge are captured using a line scan camera (e.g., the line scan camera of imaging system 112). Each image is captured at various rotational positions of the container. In the expressions used herein, it should be understood that even if an image is captured when the container has stopped, it is considered that the image was captured "while the container was spinning". For example, in some embodiments, the timing of each image capture by the line scan camera may coincide with a short time when the container is stationary (e.g., the container is stationary during discrete small rotational intervals while generally spinning the container 360 degrees in steps). Alternatively, the line scan camera can capture an image at an appropriate rotational position of the container without the container having to stop spinning / rotating at any point during the line scan. Block 806 may be performed, for example, by the line scan camera of imaging system 112 in response to a command generated by processor 120 that causes, for example, sample movement and image capture unit 134 to execute.

[0049] In block 808, a two-dimensional image of the stopper edge is generated based on at least the plurality of images. Each image captured in block 806 may provide only one (or several, etc.) pixel along the first (e.g., horizontal) axis of the two-dimensional image, but may provide all pixels along the second (e.g., vertical) axis of the two-dimensional image. Block 808 may be performed, for example, by processor 120 that causes image generation unit 136 to execute.

[0050] In block 810, an inference model including a trained neural network (e.g., neural network 500 of FIG. 5) is executed to generate output data indicating the likelihood that a sample contains a defect, thereby processing the pixels of the two-dimensional image (e.g., based on the number, size, and / or type of particles or other objects in the sample that are at or near the stopper edge). In some embodiments, block 810 includes processing the pixels of the two-dimensional image by applying intensity values associated with different pixels, or other values derived from intensity values (e.g., normalized values), to different nodes of the input layer of the trained neural network. Block 810 may be executed, for example, by processor 120 that executes inference model unit 138.

[0051] In some embodiments, method 800 includes one or more additional blocks not shown in FIG. 8.

[0052] In one embodiment, for example, method 800 includes an additional block in which a container is selectively transported to a specified rejection area based on the output data generated in block 810. This may be performed, for example, by additional conveyance means (e.g., an additional rotary table, star wheel, rail, etc. as discussed above with reference to FIG. 2) in response to a command generated by processor 120 that causes sample movement and image capture unit 134 to execute.

[0053] As another example, method 800 may be implemented in parallel with blocks 802-806 and may include blocks similar to blocks 802-806 for a second container / sample (i.e., for increasing throughput). In such an embodiment, method 800 may also include additional blocks in which an additional two-dimensional image (of the stopper edge of the second container) is generated and processed, similar to blocks 808 and 810.

[0054] While the system, method, apparatus, and their components have been described from the perspective of exemplary embodiments, the system, method, apparatus, and their components are not limited thereto. The detailed description is to be construed as illustrative only and it is not possible, and even if not impossible it is unrealistic, to describe all possible embodiments. Thus, not all possible embodiments of the present invention are described. Many alternative embodiments can be realized using either current technology or technology developed after the filing date of this patent, and such embodiments are still within the scope of the claims that define the present invention.

[0055] Those skilled in the art will understand that various modifications, changes, and combinations can be made to the above embodiments without departing from the scope of the present invention, and such modifications, changes, and combinations are to be construed as being within the scope of the concept of the present invention.

Claims

1. A method for enhancing the accuracy and efficiency in the automatic visual inspection of containers, comprising: Orienting a container containing a sample such that a line scan camera obtains a profile image of an edge of a stopper of the container; Spinning the container; While spinning the container, capturing, by the line scan camera, a plurality of images of the edge of the stopper, each image of the plurality of images corresponding to a different rotational position of the container; Generating, by one or more processors, a two-dimensional image of the edge of the stopper based at least on the plurality of images; Processing, by one or more processors executing an inference model including a trained neural network, pixels of the two-dimensional image to generate output data indicating the likelihood that the sample contains a defect; A method comprising the above.

2. Further comprising, by one or more processors, causing the container to be selectively conveyed to a designated rejection area based on the output data, the method according to claim 1.

3. The output data indicates whether the sample contains one or more objects of a specific type, the method according to claim 1.

4. The trained neural network is configured to distinguish gas-filled bubbles and particles in the sample, the method according to claim 3.

5. Processing the pixels of the two-dimensional image includes applying intensity values associated with different pixels, or other values derived from the intensity values, to different nodes of an input layer of the trained neural network, the method according to claim 1.

6. The container is a syringe, the stopper is a plunger, and the edge of the stopper is an edge of a plunger dome that contacts the sample, the method according to claim 1.

7. Orienting the container includes conveying the container using an electric rotary table or a star wheel, the method according to claim 1.

8. Orienting the container includes inverting the container such that the stopper is below the sample, the method according to claim 1.

9. Spinning the container includes rotating the container at least 360 degrees about a central axis of the container, the method according to claim 1.

10. The line scan camera is a first line scan camera, the plurality of images are a first plurality of images, the container is a first container, the two-dimensional image is a first two-dimensional image, and the method comprises: While orienting the first container, also orient the second container so that a second line scan camera obtains a profile image of the edge of the stopper of the second container; While spinning the first container, spin the second container; While capturing the first plurality of images and while spinning the second container, capture, by the second line scan camera, a second plurality of images of the edge of the stopper of the second container, wherein each image of the second plurality of images corresponds to a different rotational position of the second container; Generate a second two-dimensional image based at least on the second plurality of images; The method according to claim 1, further comprising.

11. The method according to claim 1, further comprising training the neural network using a labeled two-dimensional image of the stopper edge of the container before processing the pixels of the two-dimensional image.

12. The method according to claim 11, comprising training the neural network using labeled two-dimensional images of containers containing samples including different types, numbers, sizes, and positions of objects.

13. A line scan camera; Conveying means for orienting a container containing a sample so that the line scan camera obtains a profile image of the edge of the stopper of the container; Spinning means for spinning the container; Processing means, While the spinning means spins the container, causing the line scan camera to capture a plurality of images of the edge of the stopper, wherein each image of the plurality of images corresponds to a different rotational position of the container; Generating a two-dimensional image of the edge of the stopper of the container based at least on the plurality of images; Processing the pixels of the two-dimensional image by executing an inference model including a trained neural network to generate output data indicating whether the sample is acceptable; and An automatic visual inspection system comprising.

14. The conveying means is a first conveying means, and the automatic visual inspection system further includes a second conveying means for conveying the container to a designated rejection area, and the processing means further is configured to selectively convey the container to the designated rejection area by the second conveying means based on the output data. The automatic visual inspection system according to claim 13. **Claim 15** The output data indicates whether the sample contains one or more objects of a specific type. The automatic visual inspection system according to claim 13. **Claim 16** The trained neural network is configured to distinguish gas-filled bubbles and particles in the sample. The automatic visual inspection system according to claim 15. **Claim 17** The processing means processes the pixels of the two-dimensional image by applying at least intensity values associated with different pixels or other values derived from the intensity values to different nodes of the input layer of the trained neural network. The automatic visual inspection system according to claim 13. **Claim 18** The container is a syringe, the stopper is a plunger, and the edge of the stopper is the edge of the plunger dome that contacts the sample. The automatic visual inspection system according to claim 13. **Claim 19** The conveying means includes an electric rotary table or a star wheel, and the conveying means orients the container at least by using the electric rotary table or the star wheel to convey the container. The automatic visual inspection system according to claim 13. **Claim 20** The conveying means inverts the container so that the stopper is below the sample. The automatic visual inspection system according to claim 13. **Claim 21** While the spinning means spins the container at least 360 degrees about the central axis of the container, the processing means causes the line scan camera to capture the plurality of images. The automatic visual inspection system according to claim 13. **Claim 22** The line scan camera is a first line scan camera, the plurality of images are a first plurality of images, the container is a first container, the sample is a first sample, the conveying means is a first conveying means, the spinning means is a first spinning means, the two-dimensional image is a first two-dimensional image, and the output data is a first output data. The automatic visual inspection system further includes a second line scan camera, second conveying means, and second spinning means. The second conveying means is for orienting a second container while the first conveying means is orienting the first container, so that the second line scan camera can obtain a profile image of an edge of a stopper of the second container. The second spinning means is for spinning the second container while the first spinning means is spinning the first container. The processing means further while the first line scan camera is capturing the first plurality of images, causing the second line scan camera to capture a second plurality of images of the edge of the stopper of the second container; generating a second two-dimensional image of the edge of the stopper of the second container based at least on the second plurality of images; for processing pixels of the second two-dimensional image by executing the inference model to generate second output data indicating whether a second sample is acceptable. The automatic visual inspection system according to claim 13. [

23. ] An automatic visual inspection system, comprising a line scan camera; sample positioning hardware configured to orient a container containing a sample so that the line scan camera can obtain a profile image of an edge of a stopper of the container, and to spin the container while it is so oriented; a memory storing instructions which, when executed by one or more processors, cause the one or more processors to while the container is spinning, cause the line scan camera to capture a plurality of images of the edge of the stopper, each of the plurality of images corresponding to a different rotational position of the container. Generate a two-dimensional image of the edge of the stopper of the container based on at least the plurality of images. A memory that, by executing an inference model including a trained neural network, processes the pixels of the two-dimensional image to generate output data indicating whether the sample is acceptable. An automatic visual inspection system comprising the same.

24. The automatic visual inspection system according to claim 23, wherein the output data indicates whether the sample includes one or more objects of a specific type.

25. The automatic visual inspection system according to claim 24, wherein the trained neural network is configured to distinguish gas-filled bubbles and particles in the sample.

26. The automatic visual inspection system according to claim 23, wherein the instructions cause the one or more processors to process the pixels of the two-dimensional image by applying at least intensity values associated with different pixels or other values derived from the intensity values to different nodes of an input layer of the trained neural network.

27. The automatic visual inspection system according to claim 23, wherein the container is a syringe, the stopper is a plunger, and the edge of the stopper is the edge of a plunger dome that contacts the sample.

28. The automatic visual inspection system according to claim 23, wherein the sample positioning hardware includes an electric rotating table or a star wheel, and the container is oriented by transporting the container using at least the electric rotating table or the star wheel.

29. The automatic visual inspection system according to claim 23, wherein the sample positioning hardware inverts the container so that the stopper is below the sample.

30. The instructions cause the one or more processors to Capture the plurality of images by the line scan camera while the container spins at least 360 degrees about the central axis of the container. The automatic visual inspection system according to claim 23.

Images