Offline troubleshooting and development for automated visual inspection stations

Abstract

To provide offline troubleshooting for automated visual inspection (AVI) systems for pharmaceutical or other products.SOLUTION: In a method for replicating performance of an AVI station, a mimic AVI station that performs one or more AVI functions of the AVI station is constructed. One or more container images are captured by an imaging system while a container is illuminated by an illumination system of the AVI station, and one or more additional container images are captured by a mimic imaging system of the mimic AVI station. The method also includes, by one or more processors, identifying one or more differences between the one or more additional container images and the one or more container images, generating a visual indication of the differences and / or one or more suggestions for modifying the mimic AVI station, and modifying the mimic AVI station based on the visual indication.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] This application generally relates to an automatic visual inspection (AVI) system for pharmaceuticals or other products, and more particularly to techniques for performing offline troubleshooting and / or development for an AVI station.

Background Art

[0002] In certain situations, such as in quality control procedures for manufactured pharmaceuticals, it is necessary to examine samples (e.g., containers such as syringes or vials and / or contents such as liquid pharmaceuticals or lyophilized pharmaceuticals) for defects. The acceptability of a particular sample under applicable quality criteria may depend on metrics such as the type and / or size of container defects (e.g., chips or cracks), and / or the type, number, and / or size of undesirable particles (e.g., fibers) in the pharmaceutical. If a sample has unacceptable metrics, the sample may be determined to be non-conforming and / or may be discarded.

[0003] To handle the volumes typically associated with commercial production of pharmaceuticals, defect inspection tasks are becoming increasingly automated. Additionally, the dedicated equipment used to assist with automated defect inspection is very large, very complex, and very expensive, and significant investment in human and other resources is required to qualify and commission each new product line. As a very simple example, the Bosch® 296S commercial line equipment used in the fill-finish inspection stage of drug-filled syringes includes 15 individual visual inspection stations and has a total of 23 cameras (i.e., 1 or 2 cameras per station). This equipment is designed to detect a variety of defects, including defects related to container integrity such as large cracks in the container closure system, defects related to the surface appearance of the container such as scratches or dirt on the container surface, and defects related to the pharmaceutical itself such as the color of the liquid or the presence of foreign particles.

[0004] Purchasing additional AVI line equipment can be extremely costly, so typically, troubleshooting and new product characterization activities need to be performed on-site. Therefore, troubleshooting and new product characterization usually require long downtimes, resulting in suboptimal long-term productivity.

SUMMARY OF THE INVENTION

MEANS FOR SOLVING THE PROBLEM

[0005] The embodiments described herein relate to systems and methods in which an "emulated" AVI station is constructed or upgraded to replicate the operation of an existing AVI station, thereby enabling offline troubleshooting or new product characterization and / or qualification work that does not interfere with or minimally interferes with the operation of the production line. In some embodiments, the emulated AVI station is a dedicated offline (e.g., laboratory-based) station that emulates one or more AVI functions of an existing commercial line equipment station (e.g., one of a plurality of stations within the line equipment). In such embodiments, the emulated AVI station can be used to troubleshoot problems related to a specific corresponding station within the commercial line equipment or otherwise improve the operation of the corresponding station without requiring a long shutdown of the commercial line equipment. For example, offline modifications can be made to hardware components (e.g., lighting fixtures, star wheels, etc.), hardware arrangements (e.g., the distance or angle between the sample and the camera or lighting fixture, the configuration of the lighting fixture, etc.) and / or software (e.g., code implementing inspection algorithms). Once appropriate modifications are identified, the commercial line equipment can be shut down for a relatively short time to implement those changes in the original AVI station. Optionally, some on-site qualification work is then performed. Since the emulated AVI station is offline, it provides an opportunity to conduct root cause investigations, recipe development and / or other support activities in the laboratory rather than on the commercial line equipment.

[0006] In other embodiments, the mimic AVI station is, alternatively, a commercial line equipment station, the goal of which is to mimic the characteristics / operations of a laboratory-based AVI station. In such embodiments, a laboratory-based AVI station can be used to characterize new pharmaceuticals and qualify the testing of new pharmaceuticals, which would otherwise / conventionally require extensive downtime of the line equipment and prevent simultaneous use for other pharmaceuticals. Once appropriate hardware components / configurations and appropriate software are identified, the line equipment can be shut down for a relatively short period of time to implement those changes (where again, in some cases, subsequent qualification work on-site is then performed). Similar to the embodiments described above, this embodiment provides an opportunity to perform recipe development, root cause investigation, and / or other support activities in a laboratory rather than on commercial line equipment.

[0007] In any of these embodiments, there are significant challenges in constructing a suitably similar mimic AVI station. In particular, the imager (e.g., camera, imaging optics), lighting (e.g., lighting device, ambient / surrounding lighting or reflections), relative geometry (i.e., spatial arrangement), image processing software, computer hardware, and / or the mechanical movement of the product can all potentially affect inspection performance and it is important to closely match the replicated AVI station. This is particularly difficult because many of these components / characteristics tend to be specific to any given AVI station. Accordingly, in embodiments of the present disclosure, a robust and reliable process is used to replicate the AVI station as closely as possible (or as closely as necessary).

[0008] First, the components / configuration of the AVI station can be identified using any of a variety of suitable techniques. For example, detailed manual photographs, 3D scans, and measurements can be taken. Alternatively or additionally, for this purpose, 3-dimensional computer-aided design (CAD) files (e.g., vector image pdf or technical exploded views in other formats) can be used. This information can be used to obtain and / or assemble the hardware of the mimicked AVI station and place it in the same relative arrangement / geometry as the original AVI station (i.e., the station being mimicked). The AVI station can also be reproduced using a 3D scanner or other equipment / techniques.

[0009] Using the various techniques disclosed herein, the constructed or partially constructed mimicked AVI station can be improved and validated by comparing the sample (e.g., container) images captured by the mimicked AVI station with the sample images captured by the reproduced AVI station. The feedback obtained from this process allows the user (e.g., engineer) to not only determine whether the mimicked AVI station operates sufficiently like the original AVI station, but also to determine which aspects of the mimicked AVI station should be modified in order to better replicate the operation of the original AVI station.

[0010] In some embodiments, for this purpose, an image comparison software tool performs the comparison and generates a corresponding output. For example, the image comparison tool can calculate prominent image metrics (e.g., metrics indicating light intensity, camera noise, camera / sample alignment, focus blur, motion blur, etc.) in real time and report them to the user, providing the user with a relatively quick and reliable process for fine-tuning and evaluating the feasibility of the emulated AVI station. In some embodiments, the image comparison tool generates a specific proposal (e.g., "Reduce the distance between the camera and the container") based on the metrics, and this is displayed to the user. Advantageously, the image comparison tool can enable an accurate reproduction of the operation of the AVI station even when the original AVI station and the emulated AVI station are located remotely. That is, the operation of the AVI station can be appropriately reproduced even in certain situations where it is difficult or impossible to accurately reproduce the geometry of the hardware of the AVI station, the computer hardware, and / or other aspects. The image comparison tool generally provides a scientific and reproducible process that reduces risks associated with human error and human subjectivity, and thus is likely to convince regulatory authorities regarding the true equivalence between the AVI station and the corresponding emulated station.

[0011] Those skilled in the art will understand that the figures described herein are included for illustrative purposes and are not intended to limit the present disclosure. The drawings are not necessarily to scale and rather focus on showing the principles of the present disclosure. In some cases, it should be understood that various aspects of the described embodiments may be shown exaggerated or enlarged for ease of understanding of the described embodiments. In the drawings, like reference numerals generally refer to functionally and / or structurally similar components throughout the various drawings.

Brief Description of the Drawings

[0012]

Figure 1A

Figure 1B

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 4C

Figure 4D

Figure 5

Figure 6

Figure 7

Figure 8

DETAILED DESCRIPTION OF THE INVENTION

[0013] The various concepts introduced above and discussed in more detail below can be implemented in any of a number of ways, and the concepts described are not limited to any particular implementation. Examples of embodiments are provided for purposes of illustration.

[0014] Figures 1A and 1B show an exemplary process 100 for troubleshooting an automatic visual inspection (AVI) station of commercial line equipment by constructing and using an imitation AVI station. First, referring to Figure 1A, at stage 102, commercial line equipment equipped with one or more AVI stations operates in normal / production operation mode. The commercial line equipment can be used, for example, in the "fill-finish" stage for quality control in the manufacture of pharmaceuticals (e.g., syringes containing liquid pharmaceuticals or glass vials containing lyophilized pharmaceuticals). The AVI station can include one or more stations dedicated to container inspection (e.g., syringes, vials, etc.) and / or one or more stations dedicated to sample inspection (e.g., detection and / or characterization of particles in pharmaceuticals within a container). The commercial line equipment can be, for example, the commercial line equipment 302 discussed in more detail below in connection with Figure 3. The horizontal line / arrow at the top of Figures 1A and 1B extends from stage 102 to stage 142 (described later) and represents a continuous product line inspection using the commercial line equipment. The horizontal axis in Figures 1A and 1B generally represents time, but is not necessarily proportional to the scale of time, and Figures 1A and 1B do not necessarily represent the order of operations (although they may in some cases) (e.g., stage 110 can be performed before or after the first iteration of stage 122, etc.).

[0015] At stage 104, a problematic AVI station within the commercial line equipment is identified. For example, a person monitoring the production process may observe that a particular AVI station of the line equipment is identifying a large number of false detections (e.g., samples marked as defective by the line equipment but determined to be acceptable in more rigorous manual or automatic inspections) and / or failing to identify defective samples.

[0016] At stage 110, the operator downloads and / or installs the software code used by the problematic AVI station onto the computing system associated with the laboratory-based setup (i.e., the one that serves as the emulated AVI station). The code can be transferred directly from the line equipment or installed in another way (e.g., from a portable memory device or a download via the Internet, etc.). In some embodiments, the installed code includes code that plays a role in container movement, image capture, and image processing. For example, the code controls a mechanism that agitates (e.g., rotates, sways, inverts, etc.) the container before and / or during imaging, triggers one or more cameras when appropriate, and processes the images from the cameras to detect defects in the container (e.g., cracks, chips) and / or defects in the contents (e.g., large fibers or other foreign objects).

[0017] At stage 112, the problematic AVI station within the line equipment captures one or more images of the container. Depending on the embodiment and / or scenario, stage 112 may or may not require an interruption to the normal / production operation of the line equipment. For example, the captured images can be images that are also used during production.

[0018] At stage 114, the hardware of the problematic AVI station is reverse engineered to initiate the emulated AVI station setup procedure 120. Stage 114 may include reverse engineering of the hardware components of the problematic AVI station (e.g., cameras, optical components, lighting devices, mechanisms for moving the container, etc.), the hardware component assembly of the problematic AVI station (e.g., how various components and sub-components are assembled), the relative geometry / placement of the hardware components in the problematic AVI station (e.g., the orientation and distance of the container relative to the lighting device and camera) and / or other characteristics of the problematic AVI station (e.g., the rotational speed of the container, etc.). In some embodiments, the reverse engineering is purely manual and includes accurate (e.g., by calipers, rulers, etc.) measurements, examination of available drawings, etc. A 3D scanner can also be used to accurately capture the dimensions of the AVI station. In other embodiments, at least a portion of the reverse engineering is automated, for example, by processing a file or image indicative of the dimensions (angles, distances, etc.) of the problematic AVI station.

[0019] In the first iteration of stage 122, a mimicking AVI station is constructed using the knowledge obtained at stage 114. The construction can be partially manual or fully manual. Specific non - electronic hardware components (e.g., star wheel, etc.) of the mimicking AVI station can be constructed using any suitable fabrication techniques such as metal and plastic CNC machining and / or 3D printing. The first iteration of stage 122 may also include purchasing or otherwise obtaining various off - the - shelf components such as cameras, LED rings, or other lighting devices. The first iteration of stage 122 may also include setting various software parameters to match the parameter settings used at stage 112. For example, the user can set the rotational speed of the container to be equal to the rotational speed setting used by the line equipment when capturing an image at stage 112.

[0020] In the first iteration of stage 124, after the first attempt to replicate the problematic AVI station (at stage 122), one or more images of the container are captured by one or more imagers (e.g., cameras) of the mimicking AVI station. The container is assumed to be of the same type as the container imaged at stage 112 and may actually be the same container.

[0021] In the first iteration of stage 126, the image comparison tool determines whether the container image captured at stage 112 matches the container image captured in the first iteration of stage 124 to an acceptable extent. To make this determination, the image comparison tool may generate several metrics for each of the images or image sets and compare those metrics to determine a measure of similarity (e.g., a similarity score). For example, the image comparison tool may generate metrics related to size (e.g., the size of the container shown in the image), orientation (e.g., the angle of the container wall relative to the vertical axis of the image), light intensity (e.g., indicated by the pixel intensity of the image), focus blur, motion blur, and / or other characteristics. The image comparison tool may also compare the corresponding metrics (e.g., by calculating difference values) between the image from stage 112 and the image from stage 124. Exemplary metrics are discussed in more detail below with reference to FIGS. 6 and 7. The determination at stage 126 may be made by a user observing the output of the image comparison tool or by the tool itself, depending on the embodiment.

[0022] If the image comparison tool (or the user of the tool) determines that the image or image set is not sufficiently similar in the first iteration of stage 126, the emulated AVI station is modified in the second iteration of stage 122. The modification in the second iteration of stage 122 is intensively performed based on the output of the image comparison tool in the first iteration of stage 126. For example, if the image comparison tool indicates that the light intensity of the image captured by the emulated AVI station in the first iteration of stage 124 is too low, the user can move the lighting device closer to the container or change the aperture size of the lens, etc. during the second iteration of stage 122. As another example, if the image comparison tool indicates that the image captured in the first iteration of stage 124 is less focused than the image captured at stage 112, the user can move the container closer to or farther from the camera of the emulated AVI station. In some embodiments, the image comparison tool processes the metrics of the compared images and provides suggestions such as "move the lighting device closer to the container", "move lighting device B closer to the container", or "move lighting device B 3 mm closer to the container" at stage 126.

[0023] After the developer makes a correction during the second iteration of stage 122, the emulation AVI station captures a new set of one or more images during the second iteration of stage 124 (e.g., in response to a manual trigger from the user), and the image comparison tool compares the images captured at stage 112 (or, optionally, new images captured by the problematic AVI station) with the new images during the second iteration of stage 126. As can be seen from FIG. 1A, the loop within procedure 120 can continue for any number of iterations until the image comparison tool (or the user observing its output) determines that the emulation AVI station has reproduced the operation / characteristics of the problematic AVI station with sufficient accuracy, as indicated by a sufficient degree of similarity between the images captured by the emulation AVI station and the images captured by the problematic AVI station of the line equipment during the iteration of stage 126. In some scenarios, a sufficient degree of similarity can be achieved even when there are significant differences in the hardware components and / or geometry. In other scenarios, a sufficient degree of similarity requires an exact replication of the hardware components and geometry.

[0024] Once a sufficient degree of similarity is achieved, the emulation AVI station becomes available for use in a troubleshooting role during process 130 (see FIG. 1B). In an exemplary process 130, during the first iteration of stage 132, the user (e.g., a technician) contemplates / formulates appropriate modifications to the inspection algorithm / recipe and / or modifications to the hardware setup (e.g., different types of cameras, lenses, lighting devices, etc., and / or different arrangements and / or settings of such devices / components) in an attempt to correct the problem observed at stage 104. Thereafter, during the first iteration of stage 134, the operator modifies the hardware and / or code according to the modifications identified during the first iteration of stage 132.

[0025] In the first iteration of stage 136, use the emulation AVI station to test whether the operation of the emulation AVI station is qualified, that is, whether the problems observed at stage 104 have been sufficiently corrected. Stage 136 may include, for example, comparing standard-based requirements with statistical results (such as false detection rate, etc.). Each iteration of stage 136 may require a large number of images and / or diverse container / product samples to determine whether the problem has been solved (for example, in the case of a low but still unacceptable false detection rate or missed detection rate), so it can be time-consuming and / or labor-intensive. However, the time investment may be acceptable because it does not require interruption of the commercial line equipment.

[0026] If the operation is not determined to be qualified / acceptable at stage 136, process 130 is repeated, and new corrections are identified / formulated in the second iteration of stage 132. Process 130 can be repeated any number of times without interrupting the operation of the line equipment until the operation is determined to be qualified / acceptable in an iteration of stage 136. At that point, the troubleshooting process 130 is completed. If the qualification and commissioning activities are performed normally (at stage 140), at stage 142, the corrections made during process 130 (that is, reflected in the final state of the emulation AVI system after the final iteration of stage 134) are applied to the problematic AVI station. At stage 142, usually, it is necessary to stop production on the commercial line equipment to make changes from process 130 (and also for some simplified qualification / commissioning operations in some cases), but the downtime is significantly shorter than when it was necessary to execute the troubleshooting process 130 on-site on the problematic AVI station itself. At stage 142, after the changes are applied to the problematic AVI station, at stage 144, production (that is, the normal / production operation of the line equipment) is restarted.

[0027] Although process 100 has been described in connection with troubleshooting a problematic AVI station, it should be understood that process 100 can alternatively be used to improve an AVI station that is already operating reasonably well (e.g., to further optimize inspection performance or be more cost - efficient). Further, although process 100 has been described in connection with the fill - finish stage of a pharmaceutical manufacturing line, it should be understood that process 100 can alternatively be used at different stages (e.g., when inspecting a product after assembly of a device or when inspecting a product's indicia and / or packaging) and / or alternatively in non - pharmaceutical situations (e.g., another situation with relatively stringent quality criteria).

[0028] Figures 1A and 1B show process 100 for troubleshooting an AVI station of commercial - line equipment by constructing and using an emulated AVI station, while FIG. 2 shows an exemplary process 200 for upgrading an AVI station of commercial - line equipment to emulate the operation of a laboratory - based AVI setup. The commercial - line equipment of FIG. 2 includes one or more AVI stations and can be either, for example, the line equipment discussed above in connection with FIGS. 1A and 1B or the line equipment (i.e., commercial - line equipment 302) discussed below with reference to FIG. 3.

[0029] In stage 202, the commercial line equipment operates in the normal / production operation mode, for example, for the fill-finish stage inspection of a specific pharmaceutical (e.g., a syringe filled with a drug). As discussed above in connection with FIGS. 1A and 1B, the process 200 of FIG. 2 can alternatively be applied to different inspection stages (e.g., device assembly, packaging, etc.), and / or the process 200 can be used in non-pharmaceutical situations. The horizontal line / arrow at the top of FIG. 2 extends from stage 202 to stage 220 (described below) and represents a continuous product line inspection using the commercial line equipment. The horizontal axis in FIG. 2 generally represents time, but is not necessarily proportional to a time scale, and FIG. 2 does not necessarily represent the order of operations (however, it may represent the order of operations) (e.g., stage 202 can be performed before or after the start of stage 204).

[0030] In stage 204, it is determined to adapt the commercial line equipment for use in the fill-finish inspection of a new pharmaceutical. The new pharmaceutical may, for various reasons, require custom modifications to one or more AVI stations of the commercial line equipment. For example, the new pharmaceutical may be less transparent than previous products (e.g., requiring a stronger light intensity for imaging), or may be placed in a different type of container that may have different types of defects and / or defects in different areas.

[0031] Next, in the development procedure 210, a laboratory-based setup is used to develop an AVI station adjusted to the new pharmaceutical. Within the development procedure 210, in stage 212, the imaging system (one or more cameras and any associated optical components) of the laboratory-based setup captures an image of an illuminated sample (e.g., a liquid or lyophilized product) within a container (e.g., a syringe or vial).

[0032] At stage 214, using the captured images, a user of the laboratory-based setup develops inspection recipes / algorithms and adjusts various parameters of the laboratory-based setup to achieve desired performance (e.g., performance that is below a threshold amount of false detection and / or missed detection for a particular type of defect). The adjusted "parameters" can include any settings, types, positions, and / or other characteristics of software, imaging hardware, lighting hardware, and / or computer hardware. For example, the user can adjust the setting of the light intensity, the settings of the camera, the camera lens type or other optical components, and the geometry of the imaging system and the lighting system. It should be understood that the term "user" as used throughout this disclosure can refer to one person or a team of two or more people. In some scenarios, the user can develop an entirely new software algorithm at stage 214.

[0033] At stage 216, the user performs characterization and qualification operations to determine whether the laboratory-based setup / station developed / adjusted at stage 214 operates satisfactorily (e.g., in accordance with the applicable regulations). If it does not operate satisfactorily, the laboratory-based setup / station can be further developed / adjusted in the next iteration of stage 214, which may also require capturing additional images in the next iteration of stage 212. Stages 212, 214, and 216 of the development procedure 210 can be repeated any number of times until the results of the characterization / qualification operations at stage 216 are considered acceptable.

[0034] If the results are considered satisfactory and preparations are made to produce the new pharmaceutical on a commercial scale, the commercial line equipment is stopped and the hardware and / or software of the AVI station of the line equipment is updated at stage 220 to mimic the operation of the laboratory-based setup. Although not explicitly shown in Figure 2, the mimicking process within stage 220 may include procedures similar to the iterative setup procedure 120 of Figure 1A (i.e., stages 122, 124, 126), but may involve (possibly) the first iteration of stage 122 that does not require "construction" due to the fact that the AVI station to be updated already exists. In fact, in some scenarios, stage 122 may be skipped entirely (if the developer believes that the existing AVI station is already close enough to the laboratory-based setup to proceed directly to the image comparison tool phase) and only executed in subsequent iterations.

[0035] The developer may also need to perform some level of on-site qualification work at stage 220 that increases the downtime of the line equipment (after the update based on the image comparison tool has been successfully performed). However, the time required for this can be much shorter than the qualification work during the development procedure 210, and in any case, the fact that the development work (in procedure 210) is performed offline results in a significant reduction in the downtime of the line equipment. At stage 222, after normal qualification, production on the commercial line equipment resumes, but here it resumes for the new pharmaceutical.

[0036] In some scenarios, both process 100 and process 200 are sequentially performed. For example, if it is determined that the AVI station of the line equipment is generating an unacceptable number of false detections (i.e., flagging a significant defect when there is no such defect), process 100 can be used to construct and adjust an emulated AVI station. Thereafter, by using the emulated AVI station, it may be determined that different hardware components (e.g., an LED ring light instead of multiple directional lights) are required. After qualification of the new design on the emulated AVI station, process 200 can be used when upgrading the AVI station of the line equipment to ensure that the upgraded station exactly / sufficiently matches the operation of the emulated station.

[0037] FIG. 3 is a simplified block diagram of an exemplary system 300 that can implement the techniques described herein. Specifically, FIG. 3 shows an embodiment of using an emulated AVI station to troubleshoot the AVI station of commercial line equipment. Thus, for example, system 300 can perform and / or be used to perform process 100 of FIGS. 1A and 1B.

[0038] As can be seen from FIG. 3, the system 300 includes a commercial line device 302 and an imitation AVI station 304. The line device 302 can be a device of any production grade equipped with N (N≧1) AVI stations 310-1 to 310-N (collectively also referred to as AVI station 310). By way of example only, the line device 302 can be a Bosch® 296S line device. In the example of FIG. 3, the i-th AVI station 310-i of the line device 302 requires troubleshooting (or alternatively is a target for optimization), where i is equal to 1, N, or any number from 1 to N. Each of the AVI stations 310 serves to capture images used for inspecting containers and / or samples in the containers in different ways. For example, the first AVI station 310-1 can capture an image of a syringe or vial from above to inspect for cracks or chips, and the second AVI station 310-2 can capture an image from the side to inspect for foreign particles in the contents of the syringe or vial (e.g., liquid or lyophilized pharmaceuticals), etc.

[0039] FIG. 3 shows the general components of the i-th AVI station 310-i in the form of a simplified block diagram. In particular, the AVI station 310-i includes an imaging system 312, an illumination system 314, and sample positioning hardware 316. It should be understood that the other AVI stations 310 (if any) can be similar, but may have different component types and configurations depending on the purpose of each given station 310.

[0040] The imaging system 312 includes one or more imaging devices and optionally associated optical components (e.g., additional lenses, mirrors, filters, etc.) for capturing images of each sample (e.g., a container and a pharmaceutical). The imaging device can be, for example, a camera equipped with a charge-coupled device (CCD) sensor. As used herein, the terms "camera" or "imaging device" can refer to any suitable type of imaging device (e.g., a camera that captures the human visible frequency spectrum portion or an infrared camera, etc.). The illumination system 314 includes one or more illumination devices for illuminating each sample for imaging, such as a light-emitting diode (LED) array (e.g., a panel or a ring device).

[0041] The sample positioning hardware 316 can include any hardware for holding or otherwise supporting the sample and optionally hardware for transporting and / or otherwise moving the sample for the AVI station 310-i. For example, the sample positioning hardware 316 can include a star wheel, a turntable, a robotic arm, etc. In some embodiments, depending on the function of the AVI station 310-i, the sample positioning hardware 316 also includes hardware for agitating each sample. For example, when the AVI station 310-i inspects foreign particles in a liquid, the sample positioning hardware 316 can include components for spinning / rotating, inverting, and / or rocking each sample.

[0042] The line device 302 also includes a processing unit 320 and a memory unit 322. The processing unit 320 includes one or more processors, and each of the one or more processors may be a programmable microprocessor that executes software instructions stored in the memory unit 322 to perform some or all of the functions controlled by the software of the commercial line device 302 described herein. Alternatively or additionally, some of the processors in the processing unit 320 may be other types of processors (e.g., application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc.), and some of the functions of the processing unit 320 described herein may alternatively be implemented in hardware. The memory unit 322 may include one or more volatile memories and / or non-volatile memories. The memory unit 322 may include any one or more suitable memory types, such as read-only memory (ROM), random access memory (RAM), flash memory, solid-state drives (SSDs), hard disk drives (HDDs), etc. The memory unit 322 may collectively store one or more software applications, the data received / used by those applications, and the data output / generated by those applications.

[0043] The processing unit 320 and the memory unit 322 are configured to together control / automate the operation of the AVI station 310 and process the images captured / generated by the AVI station 310 to detect corresponding types of defects in the container and / or the contents of the container (e.g., pharmaceuticals). In an alternative embodiment, the functions of the processing unit 320 and / or the memory unit 322 are each distributed among N different processing units and / or memory units, each specific to a different one of the AVI stations 310-1 to 310-N. In yet another embodiment, some of the functions of the processing unit 320 and the memory unit 322 (e.g., for sample conveyance, agitation, and / or imaging) are distributed to the AVI station 310, while other functions of the processing unit 320 and the memory unit 322 (e.g., for processing sample images to detect defects) are performed by a centralized processing unit. In some embodiments, at least a portion of the processing unit 320 and / or the memory unit 322 is included in a computing system (e.g., a specially programmed general-purpose computer) external (and optionally remote) to the line equipment 302.

[0044] The memory unit 322 stores the sample (container / product) images captured by the AVI station 310 and also stores the AVI code 326. When the AVI code 326 is executed by the processing unit 320, it causes (1) the AVI station 310 to capture an image and (2) process the captured image (e.g., as described above) to detect defects. For example, in the case of the AVI station 310-i, the AVI code 326 includes the corresponding portion shown as code 328 in FIG. 3. As an example of one embodiment, the code 328 may trigger the imaging system 312 to capture an image while the sample is being illuminated by the illumination system 314, control the sample positioning hardware 316 to position the sample correctly at the right time, and optionally stir the sample according to a stirring profile at the appropriate time. After the image is captured and stored as image 324, the code 328 processes the image 324 to detect specific types of defects associated with the station 310-i. As described above, in some embodiments, the portion of the code 328 that processes the image may be executed by a different processor, component, and / or device than the portion of the code 328 that controls imaging, stirring, etc.

[0045] The mock AVI station 304 can be a laboratory-based setup constructed to replicate (to a sufficient extent) the operation of a particular AVI station 310-i (e.g., in response to knowing that there are unacceptable levels of false detection or missed detection at the AVI station 310-i). The mock AVI station 304 includes a mock imaging system 332, a mock illumination system 334, and mock sample positioning hardware 336. The mock imaging system 332 includes one or more imaging devices (and optionally associated optical components) for capturing images of each sample (e.g., containers and pharmaceuticals), the mock illumination system 334 includes one or more lighting devices for illuminating each sample for imaging, and the sample positioning hardware 316 includes hardware for holding or otherwise supporting the sample and optionally hardware for transporting and / or otherwise moving the sample.

[0046] Ideally, the emulation imaging system 332, the emulation illumination system 334, and the emulation sample positioning hardware 336 exactly replicate the imaging system 312, the illumination system 314, and the sample positioning hardware 316 of the AVI station 310-i, respectively. More importantly, the emulation AVI station 304 as a whole ideally replicates the operation of the AVI station 310-i. However, in the real world, it is extremely difficult to achieve an exact match of operations. As described above in connection with FIG. 1A, various manual and / or automatic reverse engineering techniques can be used to construct an emulation AVI station 304 that initially has an operation "close" to that of the AVI station 310-i (e.g., at stage 122).

[0047] As described above in connection with stage 126 of FIG. 1A, after the emulation AVI station 104 is initially constructed, software can be used to facilitate fine-tuning of the emulation AVI station 304 to improve the match of operations. For this purpose, the emulation AVI station 304 is coupled to a computing system 340 (e.g., a specially programmed general-purpose computer) including a processing unit 342 and a memory unit 344. The computing system 340 can be separate from or integral with the emulation AVI station 304, and can be near or remote to the emulation AVI station 304. In some embodiments, for example, the computing system (or a portion thereof) receives images from the emulation AVI station 304 via an Internet link.

[0048] The processing unit 342 includes one or more processors, and each of the one or more processors can be a programmable microprocessor that executes software instructions stored in the memory unit 344 to perform some or all of the functions controlled by the software of the computing system 340 described herein. Alternatively or additionally, some of the processors in the processing unit 342 can be other types of processors (e.g., ASIC, FPGA, etc.), and some of the functions of the processing unit 342 described herein can alternatively be implemented in hardware. The memory unit 344 can include one or more volatile memories and / or non-volatile memories. The memory unit 344 can include one or more any suitable memory types, such as ROM, RAM, flash memory, SSD, HDD, etc. The memory unit 344 can collectively store one or more software applications, the data received / used by those applications, and the data output / generated by those applications.

[0049] Memory unit 344 stores the image 346 captured by the emulation imaging system 332 and the image 348 captured by the imaging system 312 of the AVI station 310-i. The memory unit 344 also stores an image comparison tool (ICT) 350 and an AVI code 352. Generally, the image comparison tool 350 facilitates the process of adjusting the emulation AVI station 304 so that its operation matches that of the AVI station 310-i (as described above in relation to the stage 126 and as will be described later in relation to FIGS. 6 and 7), and the AVI code 352 is used to control the constructed and adjusted emulation AVI station 304 during the troubleshooting or optimization process. The AVI code 352 can be, for example, an exact (or very close) copy of the code 328 and can be downloaded or uploaded from the line equipment 302, a portable memory device, the Internet, or another suitable source. In other embodiments, the AVI code 352 includes only a portion of the code 328 (excluding, for example, the portion used to control the conveyance of the sample to and from the appropriate imaging positions).

[0050] The computing system 340 is coupled to an output unit 360. The output unit 360 can be any type of visual and / or audio output device (such as a computer monitor, touch screen, or other display of the computing system 340 and / or a speaker, or a separate computing device having a display and / or a speaker and coupled to the computing system 340). The image comparison tool 350 and the AVI code 352 can cause the output unit 360 to provide various visual and / or audio outputs to the developer or user of the emulation AVI station 304. For example, the image comparison tool 350 can cause the output unit 360 to display various metrics representing the difference between the image 346 and the image 348 (as will be discussed further below), and the AVI code 352 can cause the output unit 360 to display information such as an indicator of whether a particular sample is defective.

[0051] Figure 3 shows an embodiment in which the emulation AVI station (Station 304) is used to troubleshoot the AVI station of the commercial line equipment (Station 310-i of Line Equipment 302). It should be understood that similar components may be used for embodiments in which the AVI station of the commercial line equipment is updated / modified to replicate the operation of a laboratory-based setup / station (such as the process 200 in FIG. 2). In such an embodiment, the i-th AVI station 310-i is an "emulation" station that emulates the AVI station 304 (i.e., the laboratory-based setup). Further, in such an embodiment, the image comparison tool 350 may alternatively reside in the memory 322 of the line equipment 302. Alternatively, the image comparison tool 350 may remain in the computing system 340.

[0052] In some embodiments, the system 300 provides access to remote sites (e.g., global manufacturing sites) so that remote users can directly access the laboratory-based setup regardless of their respective laboratories and manufacturing locations. Such an approach enables real-time collaboration between sites / users over the network, allowing the expertise of individuals (such as technicians) in various locations to be utilized while placing troubleshooting and / or development support in a global centralized facility. Thus, the network-based approach can result in a more efficient organizational structure.

[0053] FIGS. 4 and 5 show exemplary laboratory-based setups, either of which may be the emulation AVI station 304 of FIG. 3 (e.g., the emulation AVI station of process 100) or alternatively a laboratory-based AVI station used as a development platform (e.g., the laboratory-based setup used in process 200). It should be understood that these laboratory-based setups / stations are merely exemplary and that alternative types and configurations exist substantially without limitation.

[0054] First, referring to FIG. 4A, a first laboratory-based setup 400 can be used, for example, to inspect for defects (such as cracks, chips, etc.) in the upper part of a container (e.g., the upper part of a syringe filled with a liquid pharmaceutical). The setup 400 includes a camera 402 for imaging the container and an LED ring 404 for illuminating each container while imaging it. The sample positioning hardware 406 includes a platform 406A and a star wheel 406B attached to the platform 406A. Also, the camera 402 and the LED ring 404 can be attached (directly or indirectly) to the platform 406A. The star wheel 406B can hold a container (e.g., a syringe) in a fitting along the periphery of the star wheel 406B, and the star wheel 406B can rotate relative to the platform 406A to move each container to the imaging position (i.e., the center within the LED ring 404 and directly below the camera 402). However, in some embodiments and / or scenarios, rather than the setup 400 being able to sequentially image several consecutive containers, it is only important that the characteristics (such as illumination, optics, etc.) of the line equipment AVI station match or closely approximate those for imaging a single container. Thus, in some embodiments, even if the AVI station being mimicked (or the AVI station where the development activity is taking place) requires such hardware or functionality, the star wheel 406B need not be designed to hold multiple containers and need not rotate.

[0055] FIG. 4B shows an exemplary image 450 that can be captured by the setup 400 for an embodiment in which the setup 400 (and the corresponding AVI station of the line equipment) captures an image from above the syringe to detect defects on the syringe flange. As can be seen from this example, the defects can include chips and cracks of various positions and sizes on the syringe flange.

[0056] FIG. 4C shows an exemplary image 460 representing an image captured by the AVI station of the line equipment (e.g., the AVI station 310-i of FIG. 3), and FIG. 4D shows an exemplary image 470 representing an image captured by the emulated AVI station (e.g., setup 400) after successful troubleshooting. In this example, the line equipment AVI station has an unacceptably high false detection rate due to the reflection of light seen in image 460, and modifications during troubleshooting (e.g., changing from another type of lighting device to an LED ring, or changing the orientation of the camera or lighting device with respect to the container) result in significantly less reflection seen in image 470. By reducing artifacts related to light reflection, the likelihood that defects will become invisible can be reduced, and / or the likelihood of misinterpreting the reflection as a defect can be reduced.

[0057] Next, referring to the example of FIG. 5, a laboratory-based setup 500 can be used to inspect for defects (e.g., cracks, chips, dirt, etc. in the sidewall of the container syringe, and / or unacceptable types and / or numbers of particles in the pharmaceutical) in a syringe as viewed from the side (e.g., the side of a syringe filled with a liquid pharmaceutical). Setup 500 includes a camera 502 for imaging the container and an LED backlight 504 for illuminating each container from behind while imaging each container. The sample positioning hardware includes a turntable 506 that holds several syringes and positions a single syringe between camera 502 and LED backlight 504. Turntable 506 can rotate to move each syringe to the imaging position. As described above, in some embodiments and / or scenarios, it is only important that setup 500 match or closely approximate the characteristics of the line equipment AVI station with respect to imaging a given sample, rather than having the same ability to sequentially image several consecutive samples. Thus, in some embodiments, even if the emulated AVI station (or the AVI station for which development activities are being performed) requires such hardware or functionality, turntable 506 need not be designed to hold multiple syringes and need not rotate.

[0058] FIG. 6 shows an exemplary image comparison tool 600 (e.g., the image comparison tool 350 of FIG. 3) that can be used to facilitate the construction and / or upgrade of an emulation AVI station (e.g., the laboratory-based setup in process 100 of FIG. 1, the in-line equipment AVI station of process 200 of FIG. 2, or the emulation AVI station 304 of FIG. 3). As can be seen from FIG. 6, the image comparison tool 600 receives an original image 602 and an emulation image 604, and depending on whether process 100 or process 200 is being performed, the original image 602 can be an image captured by the in-line equipment's AVI station, and the emulation image 604 can be an image captured by the laboratory-based setup, or vice versa. In some embodiments, it is contemplated that the images 602, 604 use a lossless image compression format.

[0059] The image comparison tool 600 includes a metric generation unit 612 and a feedback unit 614. The metric generation unit 612 processes images 602 and 604, generates / calculates metrics indicating the characteristics of images 602 and 604, and calculates one or more additional metrics indicating the differences between images 602 and 604 based on those metrics. In some embodiments, the metric generation unit 612 calculates metrics for each image, i.e., by comparing one of the original images 602 with one of the imitation images 604. In other embodiments, the metric generation unit 612 calculates each of the metrics based on a set of multiple original images 602 and a set of multiple imitation images 604. For example, a set of x (x>1) images of the original image 602 can be averaged or overlaid on each other, and a set of x images of the imitation image 604 can be averaged or overlaid on each other after applying an alignment technique as appropriate. By such an approach, the influence of, for example, outlier images can be reduced. Averaging / overlaying / alignment can be performed, for example, by the same computing device or processor implementing the image comparison tool 600, or by another device or processor. In an alternative embodiment (e.g., when a line scan camera is used), before the unit 612 calculates any metrics, x images of the original image 602 are stitched together (e.g., in an embodiment where each container is rotated to view the container from all directions), and x images of the imitation image 604 are stitched together. Other preprocessing of images 602 and 604 (e.g., generating an image in which each pixel has the maximum intensity over all of the x images at that pixel position) is also possible. For the sake of simplicity of explanation, in the remaining description of FIG. 6 and the description of FIG. 7, only one original image 602 and one imitation image 604 are referred to. However, it should be understood that the above variations or other suitable variations are possible. Some specific examples of metrics that the metric generation unit 612 can calculate will be described below with reference to FIG. 7.

[0060] The feedback unit 614 is configured to present one or more outputs to a user (e.g., engineer, developer, technician, etc.) in visual and / or audio form based on the differential metric generated by the metric generation unit 612. The output can be generated (e.g., displayed and / or emitted) by the output unit 360 of FIG. 3, for example. In some embodiments, the feedback unit 614 simply presents the generated metric to the user (e.g., on a graphical user interface (GUI)). For example, the feedback unit 614 can display an intensity level metric, an image focus blur metric, and / or other metrics to the user.

[0061] In other embodiments, the feedback unit 614 alternatively or additionally generates one or more user suggestions based on one or more metrics. For example, the feedback unit 614 can generate suggestions such as move the camera, move the lighting device, change the light intensity, change the camera settings, etc. Other exemplary suggestions are described below with reference to FIG. 7.

[0062] In some embodiments, the metric generation unit 612 and the feedback unit 614 operate substantially in real time when the mimic image 604 is captured by the imaging system of the mimic AVI station (e.g., the mimic imaging system 332) and / or when the mimic image 604 is received by a device or system implementing the image comparison tool 600 (e.g., by the computing system 340). Thus, for example, a user can capture a mimic station image by selecting an interactive control on a GUI presented on the output unit 360 and then can almost immediately view the corresponding metrics and / or suggestions generated by the metric generation unit 612 and / or the feedback unit 614. In this way, the user can modify the mimic AVI station (e.g., fine-tune component positions, settings, etc.) and quickly proceed with the iteration of observing the effect of the modification on the operation of the mimic AVI station (i.e., how much closer or farther the mimic AVI station gets to the operation of the original AVI station due to the modification).

[0063] FIG. 7 shows an exemplary algorithm 700 that can be implemented, for example, by the image comparison tool 600 of FIG. 6 (more specifically, the metric generation unit 612). As can be seen from FIG. 7, the algorithm 700 accepts as input a single original image 702 (e.g., one of the images 702) and a single mimic image 704 (e.g., one of the images 704). As described above, in some embodiments, these "single" images can be a composite (e.g., an average) of multiple images. In one embodiment, the algorithm 700 is implemented in Python code using the OpenCV library.

[0064] At stage 712, algorithm 700 determines one or more overall image parameters (P1 to Pj, where j ≥ 1) for the original image 702 and the mimicked image 704, and compares these parameters to check for consistency. The overall image parameters can represent relatively basic image parameters such as, for example, image size, image resolution, color depth, and / or image file format. The image comparison tool 600 can determine the overall image parameters, for example, using configuration files input to or generated by various hardware components (such as cameras) of the original AVI station and the mimicked AVI station. Algorithm 700 can cause the GUI (for example, as displayed by the output unit 360 in FIG. 3) to indicate any differences in the overall image parameters. Thus, for example, it can be easily detectable when the user has changed an image captured by a camera (or subsequent processing) in the original AVI station in some way (such as cropping, format conversion, resizing, etc.), and the mimicked AVI station can be manually reset to reproduce those image processing operations.

[0065] At stages 714A and 714B, algorithm 700 calculates one or more metrics (C1 to Ck, where k ≥ 1) for the original image 702 and the mimicked image 704, respectively. Stages 714A and 714B can assume that the overall image parameters of images 702, 704 are the same. The metrics can represent any image characteristics related to or potentially related to inspection accuracy. Also, algorithm 700 includes metrics (denoted as ΔC1 to ΔCk in FIG. 7) that indicate the differences between corresponding metrics of the two images 702, 704. The algorithm can calculate some or all of the differences, for example, by simple subtraction (such as when ΔC1 is equal to the absolute value of the difference between C1 of image 702 and C1 of image 704) or using other techniques (such as element-by-element subtraction, dot product, etc.).

[0066] As an example, the metric may include one or more light intensity metrics. The light intensity may be measured in one or more ways depending on the embodiment. For example, for each of images 702, 704, algorithm 700 may average the intensity values of all pixels to generate an average value and compare (e.g., subtract) the average values of images 702, 704. As another example, algorithm 700 may generate a pixel intensity histogram for each of images 702, 704 and then mathematically compare the histograms using known techniques. Some techniques that may be used and that result in a unary output reflecting the difference between the histograms include the Bhattacharyya distance method, the correlation method, the chi-squared method, and the intersection method. The best technique to use may vary depending on the nature of the images under consideration. Based on the histogram-based intensity analysis, algorithm 700 can take into account the dynamic range of intensities within each image (i.e., the value range between the minimum intensity value and the maximum intensity value).

[0067] In the case of complex images, the intensity metrics as discussed above may be insufficient because non-uniform illumination may be unevenly reflected from a subset of the facets of the container and its surrounding environment. Further, the coarse effects due to non-uniform ambient illumination (e.g., from windows or fluorescent ceiling lights) can result in a slow variation in the intensity across the image at large wavelengths. In some embodiments, algorithm 700 uses a low-pass frequency filter to capture such variations. Additionally or alternatively, algorithm 700 may compute a fast Fourier transform (FFT) for each of images 702, 704 and process the corresponding FFT output to determine whether the frequency components are similarly distributed for images 702, 704.

[0068] As another example, the metric may include one or more metrics indicating the blur of the image. For example, algorithm 700 may calculate the Laplacian of each of images 702, 704 to generate a univalent parameter that can be easily compared. When algorithm 700 is implemented as Python code using, for example, the OpenCV library, the blur can be calculated as follows for each of images 702, 704. Blur = cv2.Laplacian(image,cv2.CV_64F).var() Also, the Laplacian metric can also indicate the movement of the imaged container / sample relative to the imaging camera. In commercial AVI systems, it is not uncommon to move the sample quickly relative to the inspection station. For example, to achieve sharp imaging, commercial systems often employ shortening the exposure time of the camera, strobing the illumination light, moving the imaging components to track parts, or combinations of one or more of these functions. If the mimicking AVI station does not match the original AVI station in one or more of these functions, some motion blur or streaking may occur. The result is similar to blur but has a directional component. Therefore, the metric calculated using the Laplacian technique can also indicate movement. In addition to the Laplacian, primary filters such as Sobel or Prewitt can be used to obtain additional information regarding the sharpness of the image.

[0069] As another example, the metric(s) may include one or more metrics indicative of camera noise. In recent industrial digital cameras, noise may occur only in the sensor itself. Generally, signals are less likely to be corrupted after digitization. When constructing an emulated AVI station, the brand and model of the camera may, in some cases, be easily matched, so the noise levels may be similar. As an example, the cumulative noise level on a given pixel in an 8-bit grayscale image may be in the range of 0 to 10 out of a dynamic range of 0 to 255 for a particular camera brand and model. This makes the noise at room temperature negligible in most AVI applications. However, in some cases, it is conceivable that the camera noise level may affect inspection performance and / or the camera model may not be found or usable due to obsolescence. In the case of digital images, camera noise is typically effectively high frequency. Thus, algorithm 700 may generate a metric indicative of camera noise by calculating the FFT for each of images 702, 704 and processing the FFT output to produce a metric indicative of the level or relative level of the high frequency components.

[0070] As another example, the metric(s) may include one or more metrics indicative of the alignment and scaling of the imaged container (and other imaged objects such as, in some cases, portions of the sample positioning hardware). For example, algorithm 700 may determine the relative rotation of the imaged container (e.g., the angle of the vertical wall of the imaged container relative to the vertical or horizontal axis of the image itself, i.e., relative to the axis established by the camera frame) and the offset / shifts in the lateral direction and / or scale depth (determined, e.g., based on lengths and / or widths measured in pixels).

[0071] At stage 716, algorithm 700 generates a weighted comparison score for image pairs 702, 704 based on the metrics calculated at stages 714A and 714B. In particular, in this example, algorithm 700 calculates the score as score = (W1 * ΔC1) + (W2 * ΔC2) +... + (Wk * ΔCk), where ΔCi (as described above) is the difference metric corresponding to Ci for the two images 702, 704. The weights W1, W2,... Wk can represent the importance of achieving similarity for each metric, particularly with respect to which metric is more or less important to achieve equivalent AVI inspection performance. The weights can be somewhat application-specific. For example, in some embodiments or scenarios, pixel intensity levels are more important than blur, and thus the difference in intensity can be weighted more heavily than the difference in Laplacian scalar values (or other metrics indicating blur).

[0072] As described above in connection with FIG. 6, image comparison tool 600 can cause output unit 360 to display some or all of the calculated metrics (e.g., only the difference metrics ΔC1 - ΔCk or only the difference metrics that exceed a corresponding threshold) in real time. Alternatively or additionally, tool 600 can cause output unit 360 to display the score generated at stage 716. In embodiments where the score is calculated, shown, and considered sufficient (e.g., above a certain predetermined threshold), a technician or other user can determine that the emulated AVI station sufficiently matches the original AVI station and proceed to use the emulated AVI station for troubleshooting and / or optimization.

[0073] If the score is not sufficient or no score is presented and the individual metrics do not appear to be close enough, the user may analyze the displayed metrics to "fine-tune" the emulation AVI station as needed to achieve a better match. For example, if the metric indicates that the emulation image 704 is overall darker (e.g., has a lower average intensity) than the original image 702, the user can check the settings of the lighting or camera device (e.g., light intensity setting, lens aperture setting, camera gain setting, camera exposure time setting, etc.) and / or move the lighting device closer to the sample, etc.

[0074] There are several aspects related to lighting sources that can affect the intensity of image pixels. Generally, in a typical AVI application, the effect is likely to be macroscopic across the entire image. High-end industrial LEDs are commonly used in recent AVI stations. When constructing an emulation AVI station, if the LED light source is not placed at an appropriate distance from the container, the intensity may decrease. Since the intensity decreases with the square of the distance from the light source, a slight difference in positioning can have a detectable effect on the final image. Even if the position relative to the container is set, the brightness of the LED light source may still vary due to the power supply. After all other factors are excluded, the image comparison tool 600 can be used in real time to fine-tune the power of the LED light source and its subsequent brightness. Image processing techniques such as those described above provide a more accurate and subtle solution than the conventional method of using a lux meter to measure the brightness of the light source in an AVI application.

[0075] Some optical components, such as lenses, may include a manual aperture or other components that can also affect the overall image intensity (and other image characteristics such as the aperture setting that affects the sharpness of the image in some cases). These are often manual dials or screws on the lens and do not provide digital feedback. Also, these components may not have visible graduations or gridlines. To achieve highly faithful product images, telecentric lenses are commonly used in factory automatic inspection stations. These lenses include a pinhole aperture in some models that can be manually adjusted in size. This affects the amount of light that can pass through the lens and also has a characteristic effect on the sharpness of the image. Therefore, by considering and combining these two factors, the user observing the metric can appropriately set the lens characteristics related to intensity.

[0076] Also, a large and consistent decrease in intensity across the entire image may indicate that a filter (such as a polarizer or diffuser) is not correctly placed over the illumination source or in front of the camera (such as an alternative polarizer or wavelength filter). Therefore, if the user observes a large difference in the intensity of the image, they may attempt to adjust the placement of the filter.

[0077] As another example, a user examining the output of an intensity histogram and / or a low-pass frequency filter for images 702, 704 may determine whether there are significant local differences in intensity between images 702, 704. If local differences exist, the user can attempt to identify and remove the cause of those local differences by studying images 702, 704 together with the histogram and / or other metrics.

[0078] As another example, if the metric indicates that the container of the mimicking image 704 is misaligned with respect to the container of the original image 702 and / or is inappropriately scaled, the user can adjust the alignment (e.g., angle or rotation) of the container and / or the camera, the distance between the container and the camera, the zoom level of the camera (e.g., lens type or digital zoom setting), etc.

[0079] As another example, if the metric (e.g., the difference in scalar Laplacian output) indicates defocus of one image 702, 704 with respect to the other, the user can adjust the distance between the container and the camera, the motor speed, the camera exposure time, etc. Many types of telecentric lenses commonly used in inspection applications typically have too shallow a depth of field, so that even a small error in the relative placement of the lens and the container can cause the image to blur. Thus, even a small difference in distance can have a large impact on defocus. Further, as described above, some of these lenses have a variable pinhole aperture, which can also contribute to image blurring if inappropriately set. Thus, even when the metric indicates a difference in defocus, the user can adjust the aperture.

[0080] Other metrics can guide the user to adjust these and / or other aspects of the mimicking AVI station to achieve better matches, such as, for example, stray light reflections in the image, the presence of important objects in the image, the image dynamic range, image bleaching, image contrast, etc.

[0081] In some embodiments, as described above, the image comparison tool 600 generates one or more proposals based on metrics. Thus, for example, the image comparison tool 600 causes the output unit 360 to display (and / or generate a computer voice message explaining) any of the above improvement techniques, such as increasing or decreasing the distance between the camera and the container if a metric reflecting the difference in intensity and / or blur of images 702, 704 exceeds a threshold, and optionally increasing or decreasing the lens aperture.

[0082] FIG. 8 is a flowchart of an exemplary method 800 for replicating the operation of an AVI station (such as the AVI station 310-i in a troubleshooting scenario or a laboratory-based setup used for development). In method 800, at block 802, an emulated AVI station (such as the emulated AVI station 304 or AVI station 310-i in a troubleshooting scenario) that executes one or more AVI functions of the original AVI station is constructed. Block 802 can be performed manually by reverse engineering the original AVI station. However, in some embodiments, block 802 includes using software to assist in the reverse engineering process (such as by interpreting the output of a CAD file or a 3D scanner). Block 802 can include, for example, the first iteration of stage 122.

[0083] At block 804, one or more container images (i.e., images of the container at appropriate imaging positions within the original AVI station) are captured by the imaging system of the original AVI station (such as a single camera). At block 806, one or more additional container images are captured by the imaging system of the emulated AVI station (such as a single camera of the same type). Blocks 804 and 806 can be similar to stages 112 and 124 of process 100, respectively.

[0084] Thereafter, at block 808, one or more differences between the container image captured by the original AVI station and the container image captured by the emulated AVI station are identified. Block 808 may be executed by a processing unit (e.g., processing unit 342) that executes an image comparison tool (e.g., tool 350). For example, the image comparison tool can generate one or more metrics reflecting the differences at block 808. Block 808 may include, for example, stages 714A and 714B of algorithm 700 and subsequent generation of difference metrics (e.g., metrics ΔC1 to ΔCk in FIG. 7).

[0085] At block 810, a visual indication of the differences identified at block 808 is generated to assist the user in modifying the emulated AVI station. The visual indication may include, for example, one or more of the metrics calculated at block 808 (e.g., difference metrics) and / or one or more proposals based on those metrics (as discussed above in connection with FIGS. 6 and 7). Block 810 may be executed by the same processing unit that executes block 808 and may include, for example, causing an output unit (e.g., output unit 360) to present the visual indication (i.e., the metrics and / or proposals).

[0086] At block 812, the emulated AVI station is modified based on the visual indication generated at block 810. Block 812 may be executed entirely by the user (i.e., manually) or at least partially automatically (e.g., by the computing system 340 adjusting the digital settings of the camera or lighting device of the emulated AVI station). Block 812 may include, for example, the second (or subsequent) iteration of stage 122 of process 100.

[0087] Although the system, method, apparatus, and their components have been described from the perspective of exemplary embodiments, they are not limited to these exemplary embodiments. The detailed description is to be construed as illustrative only, and it is neither possible nor realistic to describe all possible embodiments. Therefore, not all possible embodiments of the present invention are described. Using either current technology or technology developed after the filing date of this patent, many alternative embodiments can be implemented, which are still included within the scope of the claims that define the present invention.

[0088] Those skilled in the art will be able to make various modifications, variations, and combinations to the above embodiments without departing from the scope of the present invention, and will understand that such modifications, variations, and combinations are construed to be within the scope of the concept of the present invention.

Claims

1. 1. A method for replicating the operation of an automated visual inspection (AVI) station, comprising: constructing an emulated AVI station to perform one or more AVI functions of the AVI station, the emulated AVI station including an emulated illumination system, an emulated imaging system, and emulated sample positioning hardware configured to hold and / or support a container containing a sample; capturing one or more container images with an imaging system of the AVI station while the container is illuminated by an illumination system of the AVI station; capturing one or more additional container images with the emulated imaging system while the container is illuminated by the emulated illumination system; determining, by one or more processors, one or more differences between the one or more additional container images and the one or more container images; generating, by the one or more processors, a visual indication of (i) the one or more differences, and / or (ii) one or more suggestions for modifying the mimicked AVI station; modifying the emulated AVI station by modifying at least the emulated illumination system, the emulated imaging system, and / or the emulated sample positioning hardware based on the visual indication; The method includes:

2. 2. The method of claim 1, wherein either (i) the AVI station is included in commercial line equipment and the mimicked AVI station is a laboratory-based setup, or (ii) the mimicked AVI station is included in commercial line equipment and the AVI station is the laboratory-based setup.

3. Modifying the mimicking station comprises: modifying the spatial arrangement of at least one illumination device of the emulated illumination system, at least one imaging device of the emulated imaging system and / or the emulated sample positioning hardware; and / or Modifying at least one illumination device of the emulated illumination system, at least one imaging device of the emulated imaging system and / or hardware components of the emulated sample positioning hardware. The method of claim 1 , comprising:

4. installing, on a computer system of said emulated AVI station, AVI software implementing inspection algorithms that are also implemented by said AVI equipment; After modifying the emulated AVI station, capturing one or more new container images with the emulated imaging system while the test container is illuminated by the emulated illumination system; determining one or more characteristics of the test container and / or a sample within the test container by processing the one or more new container images according to the inspection algorithm; The method of claim 1 further comprising:

5. 5. The method of claim 4, further comprising troubleshooting or optimizing the emulated AVI station based at least in part on the determined one or more characteristics, including modifying or further modifying the emulated illumination system, the emulated imaging system, the emulated sample positioning hardware, and / or the AVI software.

6. The method of claim 5 , further comprising modifying the AVI station in response to the modifications or further modifications to the emulated illumination system, the emulated imaging system, the emulated sample positioning hardware, and / or the AVI software.

7. the one or more container images include a first plurality of container images; the one or more additional container images include a second plurality of container images; and 2. The method of claim 1 , wherein identifying one or more differences between the one or more additional container images and the one or more container images includes identifying one or more differences between a first composite image obtained from the first plurality of container images and a second composite image obtained from the second plurality of container images.

8. Determining the one or more differences between the one or more additional container images and the one or more container images includes: Container alignment, Out of focus, strength, Intensity fluctuations, Motion blur, and / or Image sensor noise The method of claim 1 , further comprising determining a difference with respect to:

9. 2. The method of claim 1, wherein determining the one or more differences between the one or more additional container images and the one or more container images comprises: (i) calculating a Fast Fourier Transform (FFT) of the one or more container images or a composite image derived therefrom; and (ii) calculating an FFT of the one or more additional container images or a composite image derived therefrom.

10. 2. The method of claim 1, wherein identifying the one or more differences between the one or more additional container images and the one or more container images comprises generating (i) a histogram of pixel intensity levels of the one or more container images or a composite image derived therefrom, and (ii) a histogram of pixel intensity levels of the one or more additional container images or a composite image derived therefrom.

11. 2. The method of claim 1 , wherein determining the one or more differences between the one or more additional container images and the one or more container images comprises calculating (i) a Laplacian of the one or more container images or a composite image derived therefrom, and (ii) a Laplacian of the one or more additional container images or a composite image derived therefrom.

12. 2. The method of claim 1, wherein (i) identifying the one or more differences and (ii) generating the visual indication are performed in real time as the one or more additional container images are captured.

13. 2. The method of claim 1, further comprising generating, by the one or more processors, one or more suggestions for modifying the mimicked AVI station based on the one or more differences, wherein generating the visual indication comprises generating the visual indication of the one or more suggestions.

14. Generating the one or more suggestions comprises: a proposal to modify one or more configurable settings of at least one imaging device of the emulated imaging system; - a proposal to modify one or more configurable settings of at least one lighting device of said mimicking lighting system; a proposal to modify the position of at least one imaging device of the emulated imaging system; and / or Proposal for modifying the position of at least one lighting device of the mimicking lighting system. The method of claim 13, comprising generating

15. the mimic sample positioning hardware is configured to move the container according to a movement profile; and The method of claim 13 , wherein generating the one or more suggestions comprises generating suggestions to modify one or more characteristics of the movement profile.

16. receiving, by the one or more processors, one or more operating parameters of the imaging system and one or more operating parameters of the emulated imaging system; comparing, by the one or more processors, the one or more operating parameters of the imaging system to the one or more operating parameters of the mimic imaging system; generating, by the one or more processors, an additional visual indication of at least one operating parameter that differs between the imaging system and the mimicking imaging system; The method of claim 1 further comprising:

17. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: receiving one or more container images captured by an imaging system of an automated visual inspection (AVI) station; receiving one or more additional container images captured by an imitation imaging system of the imitation AVI station; determining one or more differences between the one or more additional container images and the one or more container images; generating a visual indication of (i) the one or more differences, and / or (ii) one or more suggestions for modifying the mimicked AVI station; A non-transitory computer-readable medium for causing

18. The instructions cause the one or more processors to: Container alignment, Out of focus, strength, Intensity fluctuations, Motion blur, and / or Image sensor noise 20. The non-transitory computer-readable medium of claim 17, further comprising: determining one or more differences between the one or more additional container images and the one or more container images with respect to:

19. The instructions cause the one or more processors to at least: calculating a Fast Fourier Transform (FFT) of the one or more container images or a composite image derived therefrom; calculating an FFT of said one or more additional container images or a composite image derived therefrom; 20. The non-transitory computer-readable medium of claim 17, further comprising: determining the one or more differences between the one or more additional container images and the one or more container images.

20. The instructions cause the one or more processors to at least: generating a histogram of pixel intensity levels of said one or more container images or a composite image derived therefrom; generating a histogram of pixel intensity levels of said one or more additional container images or a composite image derived therefrom; 20. The non-transitory computer-readable medium of claim 17, further comprising: determining the one or more differences between the one or more additional container images and the one or more container images.

21. The instructions cause the one or more processors to at least: calculating a Laplacian of said one or more container images or a composite image derived therefrom; calculating the Laplacian of said one or more additional container images or a composite image derived therefrom; 20. The non-transitory computer-readable medium of claim 17, further comprising: determining the one or more differences between the one or more additional container images and the one or more container images.

22. 20. The non-transitory computer-readable medium of claim 17, wherein the instructions cause the one or more processors to: (i) identify the one or more differences; and (ii) generate the visual indication in real-time as the one or more additional container images are received.

23. The instructions cause the one or more processors to generate the one or more suggestions for modifying the mimicked AVI station based on the one or more differences; and 20. The non-transitory computer-readable medium of claim 17, wherein the visual indication indicates the one or more suggestions.

24. The one or more proposals a proposal to modify one or more configurable settings of at least one imaging device of the emulated imaging system; a proposal to modify one or more configurable settings of at least one lighting device of an imitation lighting system of said imitation AVI station; a proposal to modify the position of at least one imaging device of the emulated imaging system; and / or Proposal for modifying the position of at least one lighting device of the mimicking lighting system.

24. The non-transitory computer readable medium of claim 23, comprising:

25. 1. An automated visual inspection (AVI) station, comprising: An imaging system; A lighting system; sample positioning hardware configured to hold and / or support a container containing the sample; an automated visual inspection (AVI) station including: a mimicking AVI station for performing one or more AVI functions of the AVI station, Imitation lighting system; An imitation imaging system; and a simulated sample positioning hardware configured to hold and / or support a container containing the sample. An imitation AVI station including:

1. A computing system comprising: receiving one or more container images captured by the imaging system of the AVI station; receiving one or more additional container images captured by the imitation imaging system of the imitation AVI station; determining one or more differences between the one or more additional container images and the one or more container images; generating a visual indication of (i) the one or more differences, and / or (ii) one or more suggestions for modifying the mimicked AVI station; A computing system configured to: A system including:

26. The computing system includes: Container alignment, Out of focus, strength, Intensity fluctuations, Motion blur, and / or Image sensor noise 26. The system of claim 25, configured to identify one or more differences between the one or more additional container images and the one or more container images with respect to:

27. The computing system includes at least calculating a Fast Fourier Transform (FFT) of the one or more container images or a composite image derived therefrom; calculating an FFT of said one or more additional container images or a composite image derived therefrom; 26. The system of claim 25, configured to identify the one or more differences between the one or more additional container images and the one or more container images by:

28. The computing system includes at least generating a histogram of pixel intensity levels of said one or more container images or a composite image derived therefrom; generating a histogram of pixel intensity levels of said one or more additional container images or a composite image derived therefrom; 26. The system of claim 25, configured to identify the one or more differences between the one or more additional container images and the one or more container images by:

29. The computing system includes at least applying a high pass filter to the one or more container images or a composite image derived therefrom; applying a high-pass filter to said one or more additional container images or a composite image derived therefrom; 26. The system of claim 25, configured to identify the one or more differences between the one or more additional container images and the one or more container images by:

30. The computing system includes at least calculating a Laplacian of said one or more container images or a composite image derived therefrom; calculating the Laplacian of said one or more additional container images or a composite image derived therefrom; 30. The system of claim 29, configured to identify the one or more differences between the one or more additional container images and the one or more container images by:

31. 26. The system of claim 25, wherein the computing system is configured to: (i) identify the one or more differences; and (ii) generate the visual indication in real time as the one or more additional container images are received.

32. the computing system is configured to generate the one or more suggestions for modifying the mimicked AVI station based on the one or more differences; and 26. The system of claim 25, wherein the visual indication indicates the one or more suggestions.

33. The one or more proposals a proposal to modify one or more configurable settings of at least one imaging device of the emulated imaging system; a proposal to modify one or more configurable settings of at least one lighting device of an imitation lighting system of said imitation AVI station; a proposal to modify the position of at least one imaging device of the emulated imaging system; and / or Proposal for modifying the position of at least one lighting device of the mimicking lighting system.

26. The system of claim 25, comprising:

Images