Visual Inspection System for Containers of Liquid Pharmaceutical Products

Abstract

An automated visual inspection (AVI) system may include at least one side-viewing imaging device having an optical axis passing through an object to be inspected, a proximal polarizing film axially aligned with the optical axis, a liquid crystal device axially aligned with the optical axis, a distal polarizing film axially aligned with the optical axis, and at least one light source oriented to emit illumination toward the distal polarizing film. Alternatively or additionally, the AVI system may include a side-viewing imaging device having an optical axis that enters the container through a sidewall of the container, and a ring light that is coaxially aligned with the central axis of the container, below the container, and oriented to emit light toward the bottom of the container. The AVI system may also include a bottom imaging device that is coaxially aligned with the central axis and oriented to view the bottom of the container.

Description

[Technical field]

[0001] The present application relates generally to visual inspection systems for inspection of containers of liquid pharmaceutical products, and more particularly to techniques for imaging containers / vessels of liquid pharmaceutical products without intentional agitation of the liquid. [Background technology]

[0002] In certain situations, such as quality control procedures for manufactured drug products, samples (e.g., fluid samples) need to be examined for the presence or absence of various particles (e.g., protein aggregates or debris). Under the applied quality standards, the pass or fail of a given sample may depend on metrics such as the number and / or size of undesirable particles contained within the sample. If a sample has an unacceptable metric, the sample may be rejected and discarded.

[0003] Similarly, associated containers (e.g., vials, cartridges, syringes, vessels, seals, etc.) must be inspected for the presence of various defects (e.g., scratches in a vial seal, cracks in a container, etc.) Often, various inspection systems (e.g., manual or automated visual inspection systems, etc.) are utilized to detect various defects (e.g., the presence of particles, the presence of particles that have settled to the bottom of the container, the presence of particles that are floating on the surface of the product within the container, container defects, product defects, etc.).

[0004] In order to process the volumes typically associated with commercial pharmaceutical production, particle and container inspection tasks are increasingly being automated. However, automated inspection systems have struggled to overcome various barriers to achieving good particle measurement and container fidelity without increasing system complexity. For example, liquid pharmaceutical products are often distributed in glass vials. Inspecting these glass vials for foreign objects and vial seal crimp defects is one of the most challenging challenges in the associated automated visual inspection (AVI) process. One reason this is difficult with known AVI systems is the need for agitation of the liquid to reliably detect particles. AVI systems that require agitation are highly dependent on, among other things, the fluid properties and fill levels of the liquids involved.

[0005] For example, one known method of detecting particles in a liquid-filled vial involves spinning the vial at high speed (e.g., 1000-3000 RPM) and capturing a series of images as the vial spins. Heavy particles may be hurled by centrifugal force toward the inner surface of the vial's sidewall. A silhouette of the particle may be detected from a series of images acquired from an imaging device while the vial is backlit. As the vial is rotated, the entire circumference of the vial may be inspected based on a series of images acquired from at least one stationary imaging device.

[0006] As another example, another method for detecting particles in a liquid-filled vial involves spinning the vial and abruptly stopping the spinning vial (i.e., the "spin-stop" method). Multiple images of the vial are then captured while the liquid is still moving. In the spin-stop method, for example, image data associated with subsequent images of the vial can be compared to respective image data associated with previous images of the vial to infer the presence or absence of a particle, and optionally the particle's trajectory over time.

[0007] These known techniques for detecting particles in liquid-filled vials may be suitable for detecting defects if the associated liquid is intentionally agitated. However, each method is highly dependent on several parameters such as the rotation speed of the vial, the rotation deceleration of the vial, the fluid viscosity of the liquid in the vial, the fill level of the product in the vial, the fluid surface tension of the liquid in the vial, etc. In addition, false rejects of the associated vial may result from light reflected from other imager stations in the associated AVI system due to parameters such as the rotation speed, deceleration, fluid viscosity, fill level, fluid surface tension, air bubbles, surface defects on the glass, liquid droplets formed in the neck area of ​​the vial, etc.

[0008] Agitating the liquid in the vial may improve detection of some particles, but agitating the liquid too much may result in agitation phenomena such as the formation of air bubbles in the vial, droplets that appear as cracks in the neck of the vial, etc. Due at least in part to the turnover of new product and the time required to optimize inspection parameters, known techniques for detecting particles in liquid-filled vials are not ideal for high-mix, low-volume (HMLV) production environments (e.g., clinical operations, small batches of products, etc.). Summary of the Invention [Means for solving the problem]

[0009] The embodiments described herein relate to systems and methods that improve upon conventional visual inspection techniques for containers of liquid products (e.g., pharmaceutical containers, vials, vessels, etc.). In particular, a system embodying the invention provides imaging of vessels containing liquids by capturing two-dimensional (2D) images using an automated visual inspection (AVI) system that does not purposefully rely on agitation of the liquid within the vessel.

[0010] As described herein, an AVI system may include a side-view imaging device having an optical axis passing through an at least partially translucent inspection object (e.g., a container, vessel, vial, syringe, cartridge, etc.). The inspection object is disposed at a first distance from the side-view imaging device. The AVI system may also include a proximal polarizing film axially aligned to the optical axis, disposed at a second distance from the side-view imaging device, and oriented perpendicular to the optical axis. The second distance is less than the first distance. The AVI system may further include a liquid crystal device axially aligned to the optical axis, disposed at a third distance from the side-view imaging device, and oriented parallel to the proximal polarizing film. The third distance is greater than the second distance and less than the first distance. The AVI system may still further include a distal polarizing film axially aligned to the optical axis, disposed at a fourth distance from the side-view imaging device, and oriented parallel to the proximal polarizing film and the liquid crystal device. The fourth distance is greater than the first distance. The AVI system may also include a light source oriented to emit illumination toward the distal polarizing film.

[0011] A computer-implemented method for imaging an object under inspection may include emitting illumination from a light source. The method may also include polarizing the illumination emitted from the light source using a distal polarizing film. The method may further include transmitting the polarized illumination through a liquid crystal device and through a proximal polarizing film toward the object under inspection. The method may still further include capturing an image of a sidewall of the object under inspection with a side-looking imaging device, the side-looking imaging device having an optical axis that intersects the sidewall of the object under inspection.

[0012] Alternatively or additionally, an automated visual inspection (AVI) system may include a side-viewing imager having an optical axis that passes through a sidewall of the container and into the container. The container may be at least partially translucent. The AVI system may also include a ring light that is coaxially aligned with the central axis of the container, below the container, and oriented to emit light toward the bottom of the container.

[0013] The AVI system may further comprise a holding means for supporting and / or securing the container. As described herein, the AVI system may also comprise a bottom imager coaxially aligned with the central axis and oriented to view the bottom of the container. Alternatively or additionally, the AVI system may comprise an optical axis reorienting mechanism for reorienting the optical axis of the imager relative to the central axis of the container and / or an associated light source.

[0014] A computer-implemented method for imaging a container holding a liquid sample may include illuminating the container with a ring light, the ring light being coaxially aligned with a central axis of the container, below the container, and oriented to emit light toward a bottom of the container. The method may also include capturing a side-viewing image with a side-viewing imaging device, the side-viewing imaging device having an optical axis that intersects a sidewall of the container and enters the container, and the container is at least partially translucent.

[0015] A novel method is provided for inspecting containers (e.g., vials, syringes, cartridges, etc.) for foreign objects or fibers and / or other defects (e.g., damaged crimps, flawed seals, etc.) based on captured images in a high-mix, low-volume or other manufacturing environment.

[0016] Those skilled in the art will appreciate that the drawings described herein are included for illustrative purposes and are not intended to limit the disclosure. The drawings are not necessarily to scale, with emphasis instead being placed on illustrating the principles of the disclosure. It should be understood that in some instances, various aspects of the described embodiments may be shown exaggerated or enlarged to facilitate understanding of the described embodiments. In the drawings, like reference characters throughout the various figures generally refer to functionally similar and / or structurally similar components. [Brief description of the drawings]

[0017] [Figure 1A] 1A-1C show various views of an exemplary automated visual inspection system having polarizing optical elements on either side of an inspection object and between an imaging device and a light source. [Figure 1B] Various diagrams of an exemplary automatic visual inspection system having a polarization optical element between both sides of the inspection target and an imaging device and a light source are shown. [Figure 1C] Various states of a typical liquid crystal device are shown. [Diagram 2] Another exemplary automatic visual inspection system is shown having a ring light coaxially arranged with the central axis of the container and directed to emit light toward the bottom of the container, together with an imaging device having an optical axis entering the container through the side wall of the container. [Diagram 3] A further exemplary automatic visual inspection system is shown combining the systems of FIGS. 1A, 1B, and 2 with a bottom imaging device having an optical axis coaxial with the central axis and directed to view the bottom of the container. [Figure 4] FIG. 3 shows yet another exemplary automatic visual inspection system combining the system of FIG. 3 with multiple systems of FIGS. 1A and 1B. [Figure 5A] Various exemplary container types that can be inspected using an appearance inspection system such as any of the appearance inspection systems of FIGS. 1-4 are shown. [Figure 5B] Various exemplary container types that can be inspected using an appearance inspection system such as any of the appearance inspection systems of FIGS. 1-4 are shown. [Figure 5C] Various exemplary container types that can be inspected using an appearance inspection system such as any of the appearance inspection systems of FIGS. 1-4 are shown. [Figure 6] It is a simplified block diagram of an exemplary system that can implement various techniques described herein related to the training and / or use of one or more neural networks for automatic visual inspection (AVI). [Figure 7] An exemplary method of providing an AVI system that may be similar to the AVI systems of FIGS. 1A and 1B or FIG. 2 is shown. [Figure 8] An exemplary method of providing an AVI system that may be similar to the AVI systems of FIGS. 2, 3, or 4 is shown. [Figure 9A]5 shows a bottom view of an exemplary container that may be inspected using the system of FIG. 3 or FIG. 4. [Figure 9B] 5 shows a bottom view of an exemplary container that may be inspected using the system of FIG. 3 or FIG. 4. [Figure 10A] 5 shows a bottom view of another exemplary container that may be inspected using the system of FIG. 3 or FIG. 4. [Figure 10B] 5 shows a bottom view of another exemplary container that may be inspected using the system of FIG. 3 or FIG. 4. [Figure 11A] FIG. 5 shows a side view of an exemplary container that may be inspected using any of the systems of FIGS. 1-4. [Figure 11B] FIG. 5 shows a side view of an exemplary container that may be inspected using any of the systems of FIGS. 1-4. [Figure 12A] FIG. 5 shows a side view of an exemplary container that may be inspected using any of the systems of FIGS. 1-4. [Figure 12B] FIG. 5 shows a side view of an exemplary container that may be inspected using any of the systems of FIGS. 1-4. [Figure 13A] FIG. 5 shows a side view of an exemplary container that may be inspected using any of the systems of FIGS. 1-4. [Figure 13B] FIG. 5 shows a side view of an exemplary container that may be inspected using any of the systems of FIGS. 1-4. [Figure 14A] FIG. 5 shows a side view of an exemplary container that may be inspected using any of the systems of FIGS. 1-4. [Figure 14B] FIG. 5 shows a side view of an exemplary container that may be inspected using any of the systems of FIGS. 1-4. [Figure 15] An exemplary automated visual inspection method for detecting defects in a container using the systems of FIGS. 1-4 and 6 is shown. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] The various concepts introduced above and discussed in more detail below may be implemented in any of numerous ways, and the concepts described are not limited to any particular implementation. Several example implementations are provided for illustrative purposes.

[0019] The presently disclosed automated visual inspection (AVI) system reduces the complexities associated with inspecting containers (e.g., vial 505c of FIG. 5C, cartridge 505b of FIG. 5B, syringe 505a of FIG. 5A, etc.) that contain a liquid product therein. For example, the presently disclosed AVI system can reduce, if not eliminate, variables such as container rotation speed, container deceleration rate, fluid viscosity of the product in the container, fill level of the product in the container, fluid surface tension of the product in the container, air bubbles in the container, surface defects in the glass (or plastic, etc.) of the container, droplets of liquid forming in the neck region of the container, light reflected from other imager stations in an associated AVI system, etc. Although embodiments are described herein primarily with reference to AVI systems, it should be understood that various aspects may also be applied in manual visual inspection systems.

[0020] The AVI system of the present disclosure may accommodate increased throughput speeds of associated inspection processes as compared to known systems. Additionally or alternatively, the AVI system may reduce the time required to set up an automated inspection recipe for a new product, making the AVI system particularly useful in high-mix, low-volume production scenarios (e.g., clinical operations, small batches of products, etc.). As described in certain embodiments herein, capturing images of a vial or other container without intentionally agitating the liquid product within the container may substantially eliminate the complications that varying fluid properties introduce when optimizing an associated inspection recipe.

[0021] 1A and 1B show various views of an exemplary automated visual inspection (AVI) system 100 having polarizing optical elements 115, 125 on either side of an inspection object 105 and between an imager 110 and a light source 130. An "imager" can be a camera (e.g., a CCD camera) alone or can include one or more external optical components (e.g., lenses, mirrors, etc.). As used herein, the "optical axis" of an imager refers to the axis of the optical path of the imager in the region where the optical axis passes through the object (e.g., a container) being inspected. Thus, for example, the use of mirrors can result in the "optical axis" of the imager being perpendicular to the central axis of the container, even though the imager itself is oriented to run parallel to the central axis. Multiple mirrors can be positioned around the container to combine different perspectives of the container into the resulting field of view of a single imager 110.

[0022] The AVI system 100 may include a side-viewing imaging device 110 having an optical axis 111 passing through an at least partially translucent inspection object 105. Although FIGS. 1A and 1B show that the inspection object 105 is a vial, the inspection object 105 may instead be a different type of translucent or partially translucent container (e.g., syringe 505a, cartridge 505b, etc.), or an object other than a container. The inspection object 105 is disposed at a first distance from the side-viewing imaging device 110. The AVI system 100 may also include a proximal polarizing film 115 axially aligned with the optical axis 111, disposed at a second distance from the side-viewing imaging device 110, and oriented perpendicular to the optical axis 111. The second distance is less than the first distance. The AVI system 100 may further include a liquid crystal device 120 axially aligned with the optical axis 111, disposed at a third distance from the side-viewing imaging device 110, and oriented parallel to the proximal polarizing film 115. The third distance is greater than the second distance and less than the first distance. The AVI system 100 may still further comprise a distal polarizing film 125 axially aligned with the optical axis 111 and disposed a fourth distance from the side-viewing imager 110 and oriented parallel to the proximal polarizing film 115 and the liquid crystal device 120. The fourth distance is greater than the first distance. The AVI system 100 may also comprise a light source 130 oriented to emit illumination toward the distal polarizing film 125. The light source 130 may include at least one backlight, angled illumination, or the like. As used herein, the relative terms "proximal" and "distal" refer to a relative placement with respect to an imager (e.g., the side-viewing imager 110).

[0023] As used herein, an object is "axially aligned" with a particular reference axis means that the object is positioned such that the reference axis intersects or passes through the object. With particular reference to the AVI system 100, the proximal polarizing film 115, the liquid crystal device 120, the test object 105, and the distal polarizing film 125 are axially aligned with the optical axis 111 of the side-viewing imager 110, so that light emitted from the light source 130 passes through the distal polarizing film 125, the test object 105, the liquid crystal device 120, and the proximal polarizing film 115 before being received by the side-viewing imager 110. In some embodiments, the imager 110 is not a "side-viewing" imager. For example, the elements 110, 115, and 120 may be positioned below a well containing a sample, and the elements 125 and 130 may be positioned above the well (or vice versa).

[0024] However, the AVI system 100 may be particularly useful for particle inspection in vials or other containers when using the configuration shown in Figures 1A and 1B. Although the side-looking imager 110 is shown oriented horizontally, the imager 110 may instead be tilted up or down such that the optical axis 111 of the side-looking imager 110 is not perpendicular to the central axis 106 of the container 105 being imaged. For example, multiple imagers similar to the side-looking imager 110 may be oriented at different "elevation angles" such that their associated optical axes 111 are oriented slightly up or slightly down relative to the optical axis 111 shown in Figures 1A and 1B. This may be particularly useful, for example, for generating a composite three-dimensional image of the container 105 and its contents from multiple two-dimensional images.

[0025] As shown in Figures 14A and 14B (versus Figures 13A and 13B), detection of fibers 1409 can benefit from polarizing films 115 and 125. The imaging device 110 can capture images 1400a,b of Figures 14A and 14B when the light source 130 is energized and the liquid crystal device 120 is not energized. The imaging device 110 can capture images 1300a,b of Figures 13A and 13B when both the light source 130 and the liquid crystal device 120 are energized.

[0026] 1B shows a modification of AVI system 100 by removing the input polarizing film 125 and placing a distal polarizing film 125 in front of light source 130 with vial 105 between distal polarizing film 125 and liquid crystal device 120, such that the polarizing effect can be switched on or off (by de-energizing or energizing liquid crystal device 120, respectively) to capture both filtered and unfiltered images. Thus, AVI system 100 can electronically switch polarization on and off without the use of mechanical parts.

[0027] Other types of inspections, such as inspection for crimp defects and cracks in container glass, may be adversely affected if a polarizing filter is in place, and thus the polarizing filter may be rapidly switched on and off (i.e., one image may be taken with liquid crystal device 120 powered and another image may be taken with liquid crystal device 120 not powered) so that the relevant inspections may be performed quickly with a minimal number of imagers.

[0028] FIG. 1C shows a typical constructed polarizing device 100c and shows a functional diagram. The device 100c has two polarizing films 115c, 125c on either side of a liquid crystal cell 120c, with the distal polarizing film 125c being 90° out of phase with the proximal polarizing film 115c. The charge 156c aligns the liquid crystals to maintain the same polarization of the light that was incident on the device 100c. When unpowered, the crystals rotate the light from the distal polarizing film 125c to be in phase with the proximal polarizing film 115c. In other words, when the polarizing device 100c is unpowered, the device rotates the light by 90 degrees. On the other hand, when the polarizing device 100c is powered, the crystals are aligned and do not rotate the light. The typical polarizing device 100c can be used as an "electronic shutter" since the device 100c includes polarizing films 115c, 125c on either side of the cell 120c. This allows light to pass through when no current is applied and blocks light when current is applied. Notably, the AVI system 100 of Figures 1A and 1B may represent an embodiment of a liquid crystal device 120 with the film 125 removed and replaced with a distal polarizing film 125 as shown in Figures 1A and 1B. Although the polarizing device 100c is shown as a twisted nematic device, the device 100c may include any suitable cell 120c (e.g., a smectic cell, a cholesteric cell, etc.).

[0029] The AVI system 100 is particularly useful in applications where polarization improves detection of certain types of defects, such as fibers (e.g., fiber 1409 in Figures 14A and 14B), while other types of defects (e.g., seal crimp defects, cracks in vial glass, etc.) may have low contrast against background noise (e.g., air bubbles, droplets, etc.) when the liquid crystal device 120 is not energized. The polarization angle between the light emanating from the light source and the light incident on the imager 110 can be switched between 0 degrees and 90 degrees polarization using the liquid crystal device 120.

[0030] FIG. 2 shows another exemplary AVI system 200 in which the central axis 241 of the ring light 240 is coaxially aligned with the central axis 206 of the container 205 and directed to emit light toward the bottom of the container 205. Although FIG. 2 (as well as FIGS. 3 and 4) shows the container 205 to be a vial, the container 205 may instead be a different type of translucent or partially translucent container (e.g., a syringe 505a, a cartridge 505b, a vial 505c, etc.). A side-viewing imaging device 210 has an optical axis 211 that enters the container 205 through a side wall 212 of the container 205. The AVI system 200 may further comprise a holding means (not shown in FIG. 2) for supporting and / or securing the container 205. Possible holding means are discussed in more detail below.

[0031] As used herein, an object being "coaxially aligned" with a particular reference axis means that the object is positioned such that an axis of the object (e.g., the central axis 206 of the object) is substantially aligned with (substantially the same as) the reference axis. With particular reference to the AVI system 200, by coaxially aligning the central axis 241 of the ring light 240 with the central axis 206 of the container 205, the light emitted from the ring light 240 can be projected evenly around the entire bottom and perimeter of the container 205.

[0032] The AVI system 200 is particularly suited for detecting flaw defects in vial crimps. Indeed, the ring light 240 typically set up for a bottom imager (e.g., bottom imager 335 in FIG. 3) is particularly useful for inspecting crimps when images (e.g., images 1200a, b in FIGS. 12A and 12B) are acquired from a side-view imager 210 with the ring light 240 energized. Of note, using a conventional AVI setup, the flaw defect 1109 is one of the most difficult defects to detect in a crimp 1208. For example, a conventional visual inspection system may falsely reject a container due to shading differences in the container seal (i.e., shading differences in the seal may appear as a flaw defect to a conventional AVI system). The AVI system 200 can reduce this type of false reject.

[0033] Some of the same advantages as crimp detection (e.g., inspection speed, defect clarity, etc.) also apply to particle inspection. For example, an AVI system that does not intentionally agitate the liquid in the container may simplify AVI system setup for new container types and / or new products. Additionally, an AVI system that does not intentionally agitate the liquid in the container does not rely on particle motion for particle detection. This is particularly advantageous when containers are inspected from multiple angles at the side, not just through the bottom of the container (e.g., AVI system 300 of FIG. 3). Other advantages of AVI system 200, such as less light and shadow variance, may improve identification of stationary particles.

[0034] 1A and 2, and further adds a bottom imager 335 having an optical axis 336 coaxially aligned with the central axis of the container 305 and oriented to view the bottom of the container 305. Without agitation, most particles tend to settle to the bottom of the vial and show up with good contrast in the image obtained from the bottom imager 335. The side-view imager 310 can be used to inspect for fibers and suspended particles, as well as cracks, other defects in the glass, and crimped areas.

[0035] The AVI system 300 may be faster than rotation-based inspection because there is no need to rotate the vial 305 (i.e., no ramp-up, photography, ramp-down), which can be a bottleneck in the AVI process. The AVI system 300 may alleviate bottleneck challenges and allow for closer to real-time AVI. The AVI system 300 may also result in quicker setup / programming, as no experimentation is required to determine which agitation speeds are excessive for different types of fluids / containers. The accuracy of the AVI system 300 may be quite comparable to methods involving rotation-based techniques. The AVI system 300 may detect glass and metal particles as well as fibers, for example, in vials containing liquid products.

[0036] The AVI system 300 may include a holding means 345 (e.g., a glass plate, a carousel, a star wheel, or a robotic arm that can slowly rotate the container) that supports and / or secures the container 305. The holding means 345 may also function as an optical axis redirection mechanism, which will be described in more detail herein. By combining two imagers 310, 335 with different lighting configurations (e.g., a backlight 330 and a ring light 340), most of the inspection required for the automatic visual inspection system 300 can be performed. Reducing the number of imagers and eliminating the need for agitation and fluid movement helps reduce the time it takes to set up and characterize a new product, which is typically required for HMLV operations. Object detection using such a configuration of the AVI system 300 has been found to be highly successful in detecting all particles and crimp defects. The results showed detection rates exceeding those of manual inspection: 94% for 300 μm metal particles, 100% for 1000 μm metal particles, 85% for glass particles, and 92% for fibers, all with no false rejects (i.e., good samples classified as defective).By comparison, conventional AVI equipment can require agitation combined with more than 10 imaging devices to perform an inspection.

[0037] FIG. 4 illustrates yet another exemplary AVI system 400 similar to AVI system 300, but using an additional side-viewing imaging device. In AVI system 200 or AVI system 300, some inspection processes, such as inspection of vial crimp defects, require the vial to be slowly rotated so that images around the circumference of the container can be acquired from all side perspectives. This can significantly slow down the inspection process. However, by arranging five side-viewing imaging devices 410 as in FIG. 4, the entire container 405 can be inspected without the need to rotate the container 405. Thus, a series of images can be acquired by multiple side-viewing imaging devices 410, with each imaging device 410 having a different optical axis relative to the container.

[0038] 4 illustrates a particular embodiment that alleviates the need to rotate the container 405 by disposing five imagers 410 around the perimeter of the container 405. Taking five images around the perimeter of the container 405, i.e., an image every 72 degrees, is sufficient to thoroughly inspect the container 405 for particles and cracks or chips in the container sidewall 412. However, other embodiments may include more (e.g., six) or fewer (e.g., four) side-view imagers 410. Multiple mirrors may be disposed around the container 405 to combine different perspectives of the container 405 into the resulting field of view of a single imager 410.

[0039] As can be seen in FIG. 4, the AVI system 400 may also include a bottom imager 435 that is coaxially aligned with the central axis of the container 405 and the ring light 440 and oriented to view the bottom of the container 405 .

[0040] The system 400 may include an optical axis redirecting mechanism (multiple imagers 410, each with a uniquely pointed optical axis) that redirects the optical axes relative to the sidewall 412. The optical axis redirecting mechanism (multiple imagers 410, each with a uniquely pointed optical axis) may include a container rotation device. Alternatively or additionally, the optical axis redirecting mechanism may include multiple side-viewing imagers 410 around the perimeter 405 of the container, each with a respective optical axis passing through the sidewall of the container.

[0041] 5A-5C illustrate various exemplary container types that may be used as samples imaged by visual inspection system 100 of FIGS. 1A and 1B, visual inspection system 200 of FIG. 2, visual inspection system 300 of FIG. 3, or visual inspection system 400 of FIG. 4 in a particular pharmaceutical context. Referring initially to FIG. 5A, an exemplary syringe 505a includes a hollow barrel 502, a flange 504, a plunger 506 that provides a movable fluid seal within the interior of barrel 502, and a needle shield 508 that covers the syringe needle (not shown in FIG. 5A). For example, barrel 502 and flange 504 may be formed from glass and / or plastic, and plunger 506 may be formed from rubber and / or plastic. Needle shield 508 is separated by a gap 512 by a shoulder 510 of syringe 505a. Syringe 505a contains a liquid (e.g., a pharmaceutical agent) 514 within barrel 502 and above plunger 506. Typically, the top of the liquid 514 forms a meniscus 516 with a void 518 above it.

[0042] 5B, exemplary cartridge 505b includes a hollow barrel 522, a flange 524, a piston 526 that provides a movable fluid seal within the interior of barrel 522, and a luer lock 528. For example, barrel 522, flange 524, and / or luer lock 528 may be formed from glass and / or plastic, and piston 526 may be formed from rubber and / or plastic. Cartridge 505b contains a liquid (e.g., a pharmaceutical product) 530 within barrel 522 and above piston 526. Typically, the top of liquid 530 forms a meniscus 532 with a void 534 thereover.

[0043] 5C, the exemplary vial 505c includes a hollow body 542 and a neck 544, with the transition between the two forming a shoulder 546. At the bottom of the vial 505c, the body 542 transitions into a heel 548. A crimp 550 includes a stopper (not visible in FIG. 5C) that provides a fluid seal at the top of the vial 505c, and a flip cap 552 covers the crimp 550. For example, the body 542, neck 544, shoulder 546, and heel 548 may be formed from glass and / or plastic, the crimp 550 may be formed from metal, and the flip cap 552 may be formed from plastic. The vial 505c may contain a liquid (e.g., a pharmaceutical) 554 inside the body 542. Typically, the top of liquid 554 forms a meniscus 556 (e.g., a very slightly curved meniscus if body 542 has a relatively large diameter) above which there is a gap 558. In other embodiments, liquid 554 is instead a solid material within vial 505c.

[0044] 6 is a simplified block diagram of an example system 600 that may implement various techniques described herein relating to the training (and possibly validation and / or qualification) and / or use of one or more neural networks or other machine learning (ML) systems. System 600 may also be used to test / qualify non-ML AVI systems. In addition to or as an alternative to ML systems, system 600 may include "computer vision" algorithms that do not use ML but instead use defined rules (e.g., empty vial, low fill, high fill, etc.).

[0045] FIG. 6 illustrates an embodiment in which system 600 implements one or more neural networks. Once trained and qualified, the neural networks can be used in production to detect defects associated with containers and / or the contents of those containers (e.g., the defects shown in FIGS. 9A-14B). In the pharmaceutical context, for example, the neural networks can be used to detect defects associated with syringes, cartridges, vials, or other container types (e.g., flawed crimps / seals on containers, cracks, scratches, stains, missing components, etc.) and / or to detect defects associated with liquid or lyophilized pharmaceutical products within containers (e.g., the presence or absence of fibers, metal particles, and / or other foreign particles, changes in product color, etc.). As used herein, "defect detection" can refer to classification of container images as either exhibiting or not exhibiting a defect (or a particular defect category) depending on the embodiment, and / or can refer to detection of particular objects or features (e.g., particles or cracks) that relate to whether a container and / or its contents are considered defective or not.

[0046] System 600 includes a visual inspection system (VIS) 602 communicatively coupled to a computer system 604. VIS 602 includes hardware (e.g., transport mechanisms, light sources, imagers, etc.) and firmware and / or software configured to capture digital images of a sample (e.g., a container holding a fluid or lyophilized material). VIS 602 may include, for example, any of AVI systems 100, 200, 300, 400, respectively, described herein with reference to Figures 1-4, or may be any other suitable system.

[0047] For ease of explanation, the system 600 is described herein as using container images from the VIS 602 to train and validate one or more AVI neural networks and then using the trained / validated neural networks to perform AVI / defect detection. However, it should be understood that this need not necessarily be the case. For example, the system 600 may perform training and / or validation using container images generated by several different visual inspection systems instead of or in addition to the VIS 602. Furthermore, training / validation may be performed by another system and the system 600 may then use the trained neural networks (e.g., during commercial production). In some embodiments, some or all of the container images used for training and / or validation are generated using one or more offline (e.g., laboratory-based) "mock stations" that closely replicate key aspects (e.g., optics, lighting, etc.) of commercial line equipment stations, thereby expanding the training and / or validation libraries without causing excessive downtime of the commercial line equipment.

[0048] The VIS 602 may sequentially image each of several containers. To this end, the VIS 602 may include or operate in conjunction with a holding means, such as a Cartesian robot, carousel, star wheel, and / or any other holding means, that can sequentially move each container to an appropriate position for imaging and then move the container elsewhere once imaging of the container is complete. Although not shown in FIG. 6, the VIS 602 may include a communications interface and processor that allows it to communicate with a computer system 604. In other embodiments (e.g., a laboratory-based setup), the VIS 602 includes a simpler holding means (e.g., a stage with a hole covered by a glass plate).

[0049] The computer system 604 may generally be configured to control / automate the operation of the VIS 602 and to receive and process images captured / generated by the VIS 602, as described in more detail below. The computer system 604 may be a general-purpose computer specifically programmed to perform the operations discussed herein, or may be a special-purpose computing device. As can be seen in FIG. 6, the computer system 604 comprises a user interface 606, a processing unit 610, and a memory unit 614. However, in some embodiments, the computer system 604 includes two or more computers that are co-located with each other or remote from each other. In these distributed embodiments, the operations described herein relating to the processing unit 610 and the memory unit 614 may be divided among multiple processing units and / or memory units, respectively.

[0050] The processing unit 610 includes one or more processors, each of which may be a programmable microprocessor that executes software instructions stored in the memory unit 614 to perform some or all of the functionality of the computer system 604 described herein. The processing unit 610 may include, for example, one or more graphics processing units (GPUs) and / or one or more central processing units (CPUs). Alternatively or additionally, some of the processors in the processing unit 610 may be other types of processors (e.g., application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc.), and some of the functionality of the computer system 604 described herein may instead be implemented in hardware.

[0051] The memory unit 614 may include one or more volatile and / or non-volatile memories. The memory unit 614 may include one or more of any suitable type of memory, such as a read only memory (ROM), a random access memory (RAM), a flash memory, a solid state drive (SSD), a hard disk drive (HDD), etc. Collectively, the memory unit 614 may store one or more software applications, data received / used by those applications, and data output / generated by those applications.

[0052] The memory unit 614 stores software instructions of various modules that, when executed by the processing unit 610, perform various functions for the purpose of training, validating, and / or qualifying one or more AVI neural networks. Specifically, in the exemplary embodiment of FIG. 6, the memory unit 614 includes an AVI neural network module 616 and a visual inspection system (VIS) control module 620. In other embodiments, the memory unit 614 may omit one or more of the modules 616, 620 and / or include one or more additional modules. Additionally or alternatively, one, some, or all of the modules 616, 620 may be implemented by different computer systems (e.g., remote servers coupled to the computer system 604 via one or more wired and / or wireless communication networks). Furthermore, the functionality of any one of the modules 616 and 620 may be divided among different software applications and / or computer systems. By way of example only, in an embodiment in which computer system 604 accesses a web service to train and use one or more AVI neural networks, the software instructions for AVI neural network module 616 may be stored on a remote server.

[0053] The AVI neural network module 616 includes software that trains one or more AVI neural networks using images stored in the image library 640. The image library 640 may be stored in the memory unit 614 or in another local or remote memory (e.g., memory coupled to a remote library server, etc.). In addition to training, the module 616 may implement / execute a trained AVI neural network, for example, by applying newly acquired images by the VIS 602 (or another visual inspection system) to the neural network, possibly after certain pre-processing has been performed on the images as described below. In various embodiments, the AVI neural network trained and / or executed by the module 616 may classify an entire image (e.g., defect or not, or the presence or absence of a particular type of defect, such as crimp flaws or crimp defects in general), detect objects in the image (e.g., locate foreign objects that are not air bubbles in a container image), or some combination thereof (e.g., one neural network classifies images and another performs object detection). As used herein, unless the context clearly indicates a more specific use, "object detection" refers broadly to techniques that identify the specific location of objects (e.g., particles, fibers, etc.) within an image and / or identify the specific location of features of larger objects (e.g., a flawed crimp or seal, cracks, or chips on a syringe or cartridge barrel, etc.) and can include, for example, techniques that segment (e.g., pixel-by-pixel classification) a container image or image portion, or techniques that identify objects and place bounding boxes (or other bounding shapes) around those objects.

[0054] In embodiments in which the AVI neural network detects defects in the container, the defects may relate to any suitable container characteristic. For example, with reference to the exemplary container of Figures 5A-5C, a particular AVI neural network implemented by AVI neural network module 616 may detect whether barrel 502, barrel 522, or body 542 have cracks or stains, whether flanges 504 or 524 are deformed, whether needle shield 508 is not properly positioned, whether plunger 506 or piston 526 are defective, whether luer lock 528 is defective, whether crimp 550 is properly positioned and / or defective (e.g., scratches), whether flip cap 552 is properly positioned and / or defective, etc.

[0055] Module 616 may execute the trained AVI neural network for purposes of validation, qualification, and / or testing during commercial production. In one embodiment, for example, module 616 is used only to train and validate the AVI neural network, and then the trained neural network is transferred to another computer system (e.g., using another module similar to module 616) for qualification and testing during commercial production. In some embodiments in which AVI neural network module 616 trains / executes multiple neural networks, module 616 includes separate software for each neural network.

[0056] As described above in connection with Figures 3 and 4, the ring light 340, 440 for the bottom imager 345, 435 is particularly useful for inspecting vial seal crimp defects (e.g., vial seal crimp defect 1209 in Figures 12A and 12B). For example, for 13 containers, a total of 100 images per container can be captured, resulting in a total of 1300 images. Training of the AVI neural network can be performed on images from, for example, six vials, after augmenting relevant training images by adjusting brightness, vertical mirroring, adding noise, and skewing images, as well as skewing bounding boxes (i.e., the training set can be multiplied by five). In general, deep learning can be used to detect defects in images. Using a previously trained AVI neural network further reduces the time required to set up an automated inspection recipe for a new product. The AVI neural network of the present disclosure may be implemented for high-mix, low-volume manufacturing scenarios, such as clinical operations or small batches of products, and using modern deep learning techniques (e.g., AVI neural network module 616 of FIG. 6).

[0057] In some embodiments, the VIS control module 620 controls / automates the operation of the VIS 602 such that container images may be generated with little or no human intervention. The VIS control module 620 may cause a given imager to capture a container image by sending a command or other electronic signal (e.g., generating a pulse on a control line) to that imager. The VIS 602 may send the captured container image to the computer system 604, which may store the image in the memory unit 614 for local processing. In alternative embodiments, the VIS 602 may be controlled locally, in which case the VIS control module 620 may have fewer functions than those described herein (e.g., only handles reading images from the VIS 602) or may be omitted from the memory unit 614 entirely.

[0058] FIG. 7 is an exemplary method 700 of operating an AVI system. The AVI system may be similar to, for example, the AVI system 100 of FIG. 1A and FIG. 1B. The method may include providing a side-viewing imaging device 110 having an optical axis 111 through an at least partially translucent object of inspection (e.g., vial 105), where the object of inspection is located a first distance from the side-viewing imaging device (block 702). The field of view of the side-viewing imaging device 110 may be configured to obtain a desired image of the object of inspection (e.g., image 1100a, image 1200a, image 1300a, image 1400a, etc.), for example, including the entire object of inspection or only a portion thereof. The method 700 may also include providing a proximal polarizing film 115 axially aligned with the optical axis 111, located a second distance from the side-viewing imaging device 110, and oriented perpendicular to the optical axis 111, where the second distance is less than the first distance (block 704). The method 700 may further include providing a liquid crystal device 120 axially aligned to the optical axis 111, positioned a third distance from the side-viewing imager 110, and oriented parallel to the proximal polarizing film 115, where the third distance is greater than the second distance and less than the first distance (block 708). The method 700 may still further include providing a distal polarizing film 125 axially aligned to the optical axis 111, positioned a fourth distance from the side-viewing imager 110, and oriented parallel to the proximal polarizing film 115 and the liquid crystal device 120, where the fourth distance is greater than the first distance (block 710). The method 700 may include providing a light source 130 oriented to emit illumination toward the distal polarizing film (block 712).

[0059] 8 is an exemplary method 800 of operating an AVI system. The AVI system may be similar to the AVI system 200 of FIG. 2, for example. The method 800 may include providing a side-viewing imager 210 having an optical axis 211 that enters a container (e.g., a vial 205) through a sidewall of the container 205, the container being at least partially translucent (block 802). The field of view of the side-viewing imager 210 may be configured to obtain a desired image of the inspection object (e.g., image 1100a, image 1200a, image 1300a, image 1400a, etc.), for example, including the entire inspection object or only a portion thereof. The method 800 may also include providing a ring light 240 that is coaxially aligned with the central axis 206 of the container 205, below the container, and oriented to emit light toward the bottom of the container (block 804). The method 800 may further include providing a holding means 245 for supporting and / or securing the container (block 806), as described elsewhere herein.

[0060] Figures 9A and 9B show bottom-view images 900a,b (the latter being a close-up) of an exemplary container 906 that may be inspected using the system of Figure 3 (or Figure 4 plus including a bottom imager oriented in a manner similar to imagers 335, 445, etc.). Bottom images 900a,b show a 1000 μm metal particle 907 imaged through the bottom of the container (here a vial).

[0061] Figures 10A and 10B show bottom-view images 1000a,b (the latter a close-up) of another exemplary container 1006 that may be inspected using the system of Figure 3 (or Figure 4 plus one including a bottom imager oriented in a similar manner as imager 335). Bottom images 1000a,b show a 300 μm metal particle 1007 imaged through the bottom of the container (vial).

[0062] Figures 11A and 11B show side-view images 1100a,b (the latter being a close-up) of an exemplary container 1108 that may be inspected using any of the systems of Figures 1-4. The side-view images 1100a,b show fibers 1109 imaged through the sidewall of the container (vial). Fiber contrast is improved by the use of a polarizing film (e.g., polarizing films 115, 125 as configured in Figure 1A).

[0063] Figures 12A and 12B show side-view images 1200a,b (the latter in a close-up) of another exemplary container 1208 that may be inspected using any of the systems of Figures 1-4. The side-view images 1200a,b show a flawed crimp 1209, and the boxes in Figure 12B represent the output / results of object detection performed (e.g., by AVI neural network module 616) on the images 1200a,b from the side-view imager 110 and liquid crystal device 120 with the device switched off.

[0064] 13A and 13B show side-view images 1300a,b (the latter a close-up) of a further exemplary container 1308 that may be inspected using any of the systems of Figures 1-4 without polarization effects (e.g., with liquid crystal device 120 switched on). If an image includes a droplet formed on the neck of the vial (e.g., the image of Figure 13A), an associated AVI system may erroneously identify the edge of the droplet as, for example, a crack in the associated container.

[0065] 14A and 14B show the same side-view images 1400a,b (the latter a close-up) of the same exemplary container 1408 as in FIGS. 13A and 13B, which may be inspected using any of the systems of FIGS. 1-4. With the use of a polarizing filter, the contrast ratio of the fiber 1409 to its surroundings is higher than the contrast ratio of the fiber 1309 to its surroundings. A neural network or other image processing of an AVI system is more likely to detect the fiber 1409 using the images 1400a,b of FIGS. 14A and 14B than it is to detect the fiber 1309 using the images 1300a,b of FIGS. 13A and 13B.

[0066] 15 illustrates an exemplary automated visual inspection method 1500 for detecting defects (e.g., particle 907, particle 1007, flawed crimp 1209, fiber 1309, fiber 1409, etc.) in a container (e.g., syringe 505a, cartridge 505b, vial 505c, etc.). At least a portion of method 1500 may be performed using, for example, any one of the systems of FIGS. 1-4 and 6. Method 1500 may include illuminating the container (block 1502) with a ring light (e.g., element 240, 340, or 440) positioned below the container, coaxially aligned with a central axis of the container, and oriented to emit light toward the bottom of the container. Method 1500 may also include capturing one or more side-view images of a side of the container (block 1504) using a side-view imaging device (e.g., imaging device 110, 210, 310, or 410) having an optical axis that enters the container through a sidewall of the container. The field of view of the side-view imaging device may be configured to obtain a desired image of the inspected object (e.g., image 1100a, image 1200a, image 1300a, image 1400a, etc.).

[0067] Method 1500 may further include capturing one or more bottom images of the bottom of the container using a bottom imager (e.g., imager 335 or 435) coaxially aligned with the central axis of the container (block 1508). The field of view of the bottom imager may be configured to obtain a desired image of the container (e.g., image 900a, image 1000a, etc.).

[0068] Method 1500 may also include analyzing (block 1510) one or more side view images (e.g., image 1100a, image 1200a, image 1300a, image 1400a, etc.) and / or one or more bottom images (e.g., image 900a, image 1000a, etc.) of the container using one or more processors (e.g., processing unit 610 of FIG. 6 when executing AVI neural network module 616) to detect defects. The processor may implement one or more machine learning models (e.g., classification and / or object detection models) to detect defects. For example, the processor may implement a classification model that classifies the container as "pass" or "fail." Additionally or alternatively, the processor may implement multiple machine learning models to classify a particular type of defect (e.g., a first machine learning model that classifies the defect as a fiber in the container, a second machine learning model that classifies the defect as a non-fiber particle in the container, a third machine learning model that classifies the defect as a crimp flaw, etc.).

[0069] Although the systems, methods, devices, and components thereof have been described in terms of exemplary embodiments, they are not limited to these exemplary embodiments. The detailed description is to be construed as an example only and does not describe all possible embodiments of the invention, since describing all possible embodiments of the invention would be impractical, if not impossible. Many alternative embodiments can be implemented using either current technology or technology developed after the filing date of this patent, and still fall within the scope of the claims that define the invention.

[0070] It should be understood by those skilled in the art that various modifications, variations, and combinations can be made to the above-described embodiments without departing from the scope of the present invention, and such modifications, variations, and combinations should be construed as being within the scope of the concept of the present invention.

Claims

1. A side-view imaging device having an optical axis that enters the container through the side wall of the container, wherein the container is at least partially translucent, and the side-view imaging device is positioned at a first distance from the side-view imaging device, A ring light is aligned coaxially with the central axis of the container, located below the container, and directed to emit light toward the bottom of the container. A holding means for supporting and / or fixing the container An automated visual inspection system equipped with the following features.

2. The system according to claim 1, further comprising a container rotating device, or at least one of one or more additional side-view imaging devices, each having an optical axis directed to view at least a portion of each side of the container.

3. The system according to claim 2, wherein the one or more additional side-view imaging devices consist of four side-view imaging devices.

4. The system according to any one of claims 1 to 3, further comprising a bottom imaging device that is coaxially aligned with the central axis and directed to view the bottom of the container.

5. A proximal polarizing film that is axially aligned with the optical axis, positioned at a second distance from the side-view imaging device, and oriented perpendicular to the optical axis, wherein the second distance is shorter than the first distance, A liquid crystal device that is axially aligned with the optical axis, positioned at a third distance from the side-view imaging device, and oriented parallel to the proximal polarizing film, wherein the third distance is longer than the second distance and shorter than the first distance. A distal polarizing film that is axially aligned with the optical axis, positioned at a fourth distance from the side-view imaging device, and oriented parallel to the proximal polarizing film and the liquid crystal device, wherein the fourth distance is longer than the first distance, A light source directed to emit light toward the distal polarizing film and The system according to any one of claims 1 to 3, further comprising the above.

6. The system according to any one of claims 1 to 3, further comprising a container rotating device.

7. A method for imaging a container that is at least partially translucent and holds a liquid sample, wherein the method is Illuminating the container with a ring light, wherein the ring light is aligned coaxially with the central axis of the container, is located below the container, and is directed to emit light toward the bottom of the container. The acquisition of one or more side-view images by a side-view imaging device, wherein the side-view imaging device has an optical axis that passes through the side wall of the container and enters the container, One or more bottom images are acquired by a bottom imaging device that is coaxially aligned with the central axis and directed to view the bottom of the container. Methods that include...

8. The method according to claim 7, further comprising analyzing one or more side view images of the container with one or more processors to detect at least one defect relating to the container and / or the contents of the container.

9. The method according to claim 8, wherein the at least one defect includes particles or fibers in the container.

10. The method according to claim 8, wherein the at least one defect includes a damaged container seal.

11. The method according to any one of claims 7 to 10, further comprising analyzing one or more bottom images of the container in order to detect at least one defect relating to the container and / or the contents of the container using one or more processors.

12. The method according to claim 11, wherein the at least one defect includes particles or fibers in the container.

13. The method according to any one of claims 7 to 10, further comprising analyzing one or more side view images of the container with one or more processors to classify at least one defect relating to the container and / or the contents of the container.

14. The method according to claim 13, wherein the at least one defect relating to the container and / or the contents of the container is classified as one of particles in the container, fibers in the container, or a damaged container seal.

15. The method according to any one of claims 7 to 10, further comprising analyzing one or more bottom images of the container in order to classify at least one defect relating to the container and / or the contents of the container using one or more processors.

16. The method according to claim 15, wherein the at least one defect relating to the container and / or the contents of the container is classified as particles in the container or fibers in the container.

17. The method according to any one of claims 7 to 10, further comprising analyzing one or more side view images of the containers by one or more processors to classify the containers as either pass or fail.

18. The method according to any one of claims 7 to 10, further comprising analyzing one or more bottom images of the containers by one or more processors to classify the containers as either pass or fail.

19. The method according to any one of claims 7 to 10, wherein the container is a vial.

Images