Offline troubleshooting and development of automated vision inspection stations

Abstract

In a method for replicating the performance of an automated visual inspection (AVI) station, a simulated AVI station is constructed that performs one or more AVI functions of the AVI station. One or more container images are captured by an imaging system of the AVI station while the container is illuminated by the illumination system of the AVI station, and one or more additional container images are captured by a simulated imaging system of the simulated AVI station. The method also includes: identifying, by one or more processors, one or more differences between the one or more additional container images and the one or more container images; generating, by the one or more processors, a visual indication of the difference(s) and / or one or more suggestions for modifying the simulated AVI station; and modifying the simulated AVI station based on the visual indication.

Description

Technical Field

[0001] The present application relates generally to automated visual inspection (AVI) systems for pharmaceutical or other products, and more particularly to techniques for performing offline troubleshooting and / or development of AVI stations. Background Art

[0002] In some cases (such as quality control procedures for finished drug products), it is necessary to inspect samples (e.g., containers such as syringes or vials, and / or their contents, such as liquid or lyophilized drug products) for defects. Under applicable quality standards, the acceptability of a particular sample may depend on criteria such as the type and / or size of vessel defects (e.g., nicks or cracks), or the type, number, and / or size of undesirable particles (e.g., fibers) within the drug product. If a sample has unacceptable criteria, it may be rejected and / or discarded.

[0003] To handle the volumes typically associated with commercial production of pharmaceuticals, defect inspection tasks have become increasingly automated. Furthermore, specialized equipment used to assist with automated defect inspection has become very large, very complex, and very expensive, requiring significant investment in manpower and other resources to qualify and commission each new product line. As just one example, a complete fill inspection stage for syringes filled with medications may require a significant investment in manpower and other resources to qualify and commission each new product line. The 296S commercial production line equipment includes 15 individual visual inspection stations and a total of 23 cameras (i.e., one or two cameras per station). Overall, this equipment is designed to detect a wide range of defects, including vessel integrity defects such as large cracks or vessel closure, aesthetic vessel defects such as scratches or stains on the vessel surface, and defects associated with the pharmaceutical product itself, such as liquid color or the presence of foreign particles.

[0004] Because purchasing additional pieces of AVI line equipment can be costly, troubleshooting and new product characterization activities often must be performed on-site. Consequently, troubleshooting and new product characterization often require significant downtime, resulting in suboptimal long-term production rates. Summary of the Invention

[0005] The embodiments described herein relate to systems and methods in which a "simulated" AVI station is constructed or upgraded in an effort to replicate the performance of an existing AVI station, thereby allowing offline troubleshooting or new product characterization and / or qualification work that does not interfere with, or interferes with, production line operations to a lesser extent. In some embodiments, a simulated AVI station is a dedicated offline (e.g., laboratory-based) station that simulates one or more AVI functions of a station in an existing commercial production line equipment (e.g., one of multiple stations in the production line equipment). In such embodiments, the simulated AVI station can be used to troubleshoot problems at a specific corresponding station in the commercial production line equipment, or otherwise improve the performance of the corresponding station, without requiring prolonged shutdown of the production line equipment. For example, offline modifications can be made to hardware components (e.g., lighting devices, star wheels, etc.), hardware arrangements (e.g., the distance or angle between the sample and the camera or lighting device, the configuration of the lighting device, etc.), and / or software (e.g., code implementing an inspection algorithm). Once the appropriate modifications are identified, the commercial production line equipment can be shut down for a relatively short period of time to implement these changes to the original AVI station, possibly followed by a certain amount of on-site qualification work. Because the analog AVI station is offline, it provides the opportunity to conduct root cause investigations, recipe development, and / or other support activities in the lab rather than on commercial production line equipment.

[0006] In other embodiments, the simulation AVI station is the station of commercial production line equipment instead, and target is to simulate the characteristic / performance of the AVI station based on the laboratory.In such embodiment, this AVI station based on the laboratory can be used for characterizing and evaluating the inspection of new drug product, which otherwise would / traditionally require a large amount of downtime of production line equipment and prevent it from being used for other drug products simultaneously.Once identifying suitable hardware components / configuration and suitable software, the relatively short time of production line equipment shutdown can be used to implement these changes (equally, the on-site identification work of a certain amount may be carried out subsequently).Similar to the previous embodiment, this embodiment provides the opportunity of carrying out formulation development, root cause investigation and / or other support activities in the laboratory rather than on commercial production line equipment.

[0007] In any of these embodiments, the construction of a suitably similar simulated AVI station presents significant challenges. In particular, it is important that the imager(s) (e.g., camera(s), imaging optics), illumination (e.g., lighting(s), ambient / surrounding lighting or reflections), relative geometry (i.e., spatial arrangement), image processing software, computer hardware, and / or mechanical movement of the product (all of which may affect inspection performance) closely match the AVI station being reproduced. This is particularly challenging because many of these components / features are often unique to any given AVI station. Therefore, in embodiments of the present disclosure, a robust and reliable process is used to replicate the AVI station as closely as possible (or as closely as desired).

[0008] Initially, any one of various suitable technologies can be used to identify the components / configuration of the AVI station. For example, detailed manual photography, 3D scanning and measurement can be performed. Alternatively or additionally, a three-dimensional computer-aided design (CAD) file (for example, a decomposition technical diagram in vector graphics pdf or other formats) can be used for this purpose. Utilizing this information, the hardware of the simulated AVI station can be obtained and / or assembled, and the hardware can be placed in the same relative arrangement / geometry as the original AVI station (that is, the station being simulated). 3D scanners or other equipment / techniques can also be used to recreate the AVI station.

[0009] The various techniques disclosed herein can be used to improve and validate a constructed or partially constructed simulated AVI station by comparing sample (e.g., container) images captured by the simulated AVI station with sample images captured by the AVI station being reproduced. The feedback obtained from this process can enable a user (e.g., an engineer) to determine not only whether the simulated AVI station performs in a manner sufficiently similar to the original AVI station, but also to determine which aspects of the simulated AVI station should be modified to better replicate the performance of the original AVI station.

[0010] In some embodiments, for this purpose, an image comparison software tool performs comparison and generates corresponding output. For example, the image comparison tool can calculate and report significant image indicators (e.g., indicators indicating light intensity, camera noise, camera / sample alignment, defocus, motion blur, etc.) to the user in real time, thereby providing the user with a reliable process for fine-tuning and evaluating the feasibility of the simulated AVI station in a relatively fast manner. In some embodiments, the image comparison tool generates specific suggestions based on the indicators (e.g., "reduce the distance between the camera and the container"), which are displayed to the user. Advantageously, even when the original AVI station and the simulated AVI station are located at remote locations, the image comparison tool can also achieve accurate reproduction of the AVI station performance. That is, even in certain cases where it is difficult or impossible to accurately reproduce hardware geometry, computer hardware and / or other aspects of the AVI station, the AVI station performance can also be fully reproduced. The image comparison tool generally provides a scientific, repeatable process that reduces the risk associated with human error and subjectivity, and is therefore more likely to meet the requirements of regulatory agencies for true equivalence between the AVI station and the corresponding simulated station. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The skilled artisan will understand that the drawings described herein are included for illustrative purposes and are not intended to limit the present disclosure. The drawings are not necessarily drawn to scale, with emphasis placed on illustrating the principles of the present disclosure. It should be understood that in some cases, various aspects of the described embodiments may be shown enlarged or magnified to aid in understanding the described embodiments. In the drawings, similar reference numerals generally refer to functionally similar and / or structurally similar components throughout the various figures.

[0012] Figure 1A and Figure 1B Depicts an example process for troubleshooting an AVI station for commercial production line equipment by building and using a simulated AVI station.

[0013] Figure 2 Depicted is an example process for developing AVI recipes and / or hardware settings for use in an AVI station of commercial production line equipment by simulating a laboratory-based setup.

[0014] Figure 3 It is possible to implement Figure 1A and Figure 1B A simplified block diagram of an example system for the process.

[0015] Figures 4A to 4D An example lab-based setup of a development platform that can simulate or function as an AVI station for production line equipment, and associated container images are depicted.

[0016] Figure 5Another example lab-based setup is depicted that can simulate or serve as a development platform for an AVI station of production line equipment.

[0017] Figure 6 Depicted is an example image comparison tool that may be used to facilitate construction or upgrading of a simulated AVI station.

[0018] Figure 7 Describes the Figure 6 An example algorithm implemented by the image comparison tool.

[0019] Figure 8 is a flow chart of an example method for replicating the performance of an AVI station of laboratory-based or production line equipment. DETAILED DESCRIPTION

[0020] The various concepts introduced above and discussed in greater detail below can be implemented in any of a variety of ways, and the concepts described are not limited to any particular implementation. For illustrative purposes, examples of implementation are provided.

[0021] Figure 1A and Figure 1B An example process 100 is depicted for troubleshooting an automated visual inspection (AVI) station for commercial production line equipment by building and using a simulated AVI station. Figure 1A , at stage 102, commercial production line equipment including one or more AVI stations is running in a normal / production mode of operation. For example, the commercial production line equipment can be used for quality control of the production of pharmaceutical products (e.g., syringes containing liquid pharmaceutical products or glass vials containing lyophilized pharmaceutical products) at a "fill complete" stage. The AVI station(s) can include one or more stations dedicated to container inspection (e.g., syringes, vials, etc.), and / or one or more stations dedicated to sample inspection (e.g., detecting and / or characterizing particles in a pharmaceutical product within a container). For example, the commercial production line equipment can be a station described below with reference to Figure 3 The commercial production line equipment 302 is discussed in more detail. Figure 1A and Figure 1B The top horizontal line / arrow in FIG extending from stage 102 to stage 142 (discussed below) represents uninterrupted production line inspection using commercial production line equipment. Figure 1A and Figure 1B The horizontal axis of the graph generally represents time, but is not necessarily drawn to scale, and Figure 1A and Figure 1B The order of operations is not necessarily (but may be) indicated (eg, stage 110 may occur before or after the first iteration of stage 122, etc.).

[0022] At stage 104, problematic AVI stations within the commercial production line equipment are identified. For example, an individual monitoring the production process may observe that a particular AVI station of the production line equipment is identifying a large number of false positives (e.g., samples that were marked as defective by the production line equipment but were determined to be acceptable upon closer manual or automated inspection) and / or is failing to identify defective samples.

[0023] At stage 110, the operator downloads and / or installs the software code for the AVI station in question to a computing system associated with a laboratory-based setup (i.e., it will be a simulated AVI station). The code can be transferred directly from the production line equipment, or can be installed in another manner (e.g., downloaded from a portable memory device or the Internet, etc.). In some embodiments, the installed code includes code responsible for container movement, image capture, and image processing. For example, the code can control a mechanism for shaking (e.g., rotating, shaking, inverting, etc.) the container before and / or during imaging, trigger one or more cameras at the appropriate time, and process the camera images to detect defects in the container (e.g., cracks, gaps) and / or defects in the contents (e.g., large fibers or other foreign matter).

[0024] At stage 112, the problematic AVI station within the production line equipment captures one or more images of the container. Depending on the embodiment and / or scenario, stage 112 may or may not require any interruption to the normal / production operation of the production line equipment. For example, the captured images may be images also used during production.

[0025] At stage 114, the hardware of the problematic AVI station is reverse engineered to initiate the simulated AVI station setup program 120. Stage 114 may include reverse engineering the hardware components of the problematic AVI station (e.g., camera, optics, lighting, mechanism for moving the container, etc.), the assembly of the hardware components of the problematic AVI station (e.g., how the various components and subcomponents are assembled), the relative geometry / arrangement of the hardware components in the problematic AVI station (e.g., the orientation and distance of the container relative to the lighting device(s) and camera(s), and / or other characteristics of the problematic AVI station (e.g., container rotation speed, etc.). In some embodiments, the reverse engineering is purely manual and involves precise (e.g., calipers, rulers, etc.) measurements, review of available schematics, etc. A 3D scanner may also be used to accurately capture the dimensions of the AVI station. In other embodiments, at least a portion of the reverse engineering is automated, for example, by processing a file or image indicating the dimensions (angles, distances, etc.) of the problematic AVI station.

[0026] In the first iteration of stage 122, the knowledge gained in stage 114 is used to build a simulated AVI station. This construction can be partially or completely manual. Certain non-electronic hardware components (e.g., star wheels, etc.) of the simulated AVI station can be built using any suitable manufacturing technology (such as CNC machining of metals and plastics, and / or 3D printing). The first iteration of stage 122 can also include purchasing or otherwise obtaining various off-the-shelf components, such as cameras, LED rings or other lighting devices, etc. The first iteration of stage 122 can also involve setting various software parameters to match the parameter settings used in stage 112. For example, the user can set the container rotation speed to be equal to the rotation speed setting used by the production line equipment when capturing (multiple) images in stage 112.

[0027] In a first iteration of stage 124, after initially attempting to replicate the problematic AVI station (at stage 122), one or more images of a container are captured by one or more imagers (e.g., cameras) of the simulated AVI station. The container should be of the same type as the container imaged at stage 112, and may in fact be the same container.

[0028] In a first iteration of stage 126, the image comparison tool determines whether the (multiple) container images captured in stage 112 match the (multiple) container images captured in the first iteration of stage 124 to some acceptable degree. To make this determination, the image comparison tool can generate multiple metrics for each image or set of images and compare these metrics to determine a measure of similarity (e.g., a similarity score). For example, the image comparison tool can generate metrics related to size (e.g., how large the container appears within the image), orientation (e.g., the angle of the container wall relative to the vertical axis of the image), light intensity (e.g., as indicated by image pixel intensity), defocus, motion blur, and / or other characteristics. The image comparison tool can also compare corresponding metrics of the (multiple) images from stage 112 with corresponding metrics of the (multiple) images from stage 124 (e.g., by calculating differences). In reference below Figure 6 and Figure 7 Example metrics are discussed in further detail.Depending on the embodiment, the determination at stage 126 may be made by a user observing the output of the image comparison tool or by the tool itself.

[0029] If the image comparison tool (or the user of the tool) determines in the first iteration of stage 126 that the image or set of images is not sufficiently similar, the simulated AVI station is modified in a second iteration of stage 122. The modifications in the second iteration of stage 122 are made in a focused manner based on the output of the image comparison tool in the first iteration of stage 126. For example, if the image comparison tool indicates that the light intensity of the image(s) captured by the simulated AVI station in the first iteration of stage 124 is too low, the user can move the lighting device closer to the container, change the lens aperture, etc., during the second iteration of stage 122. As another example, if the image comparison tool indicates that the image captured in the first iteration of stage 124 is less focused than the image captured in stage 112, the user can move the container closer to or further away from the camera of the simulated AVI station. In some embodiments, the image comparison tool processes the indicators of the compared images to provide suggestions in stage 126, such as "Move lighting device A closer to the container," "Move lighting device B closer to the container," or "Move lighting device B 3 mm closer to the container," etc.

[0030] After the developer makes the modification(s) in the second iteration of stage 122, the simulated AVI station captures a new set of one or more images in a second iteration of stage 124 (e.g., in response to a manual trigger by the user), and the image comparison tool compares the new image(s) with the images captured in stage 112 (or possibly with new images captured by the problematic AVI station) in a second iteration of stage 126. Figure 1A The loop within (shown) can continue for any number of iterations until the image comparison tool (or a user observing its output) determines in an iteration of stage 126 that the simulated AVI station has sufficiently accurately reproduced the performance / characteristics of the problematic AVI station, as indicated by sufficient similarity between the image(s) captured by the simulated AVI station and the image(s) captured by the problematic AVI station of the production line equipment. In some scenarios, sufficient similarity can be achieved even if there are significant differences in hardware components and / or geometry. In other scenarios, sufficient similarity requires exact duplication of hardware components and geometry.

[0031] When sufficient similarity is achieved, in process 130 (e.g. Figure 1BDuring the first iteration of stage 132 of the exemplary process 130, the simulated AVI station is prepared for troubleshooting capabilities. In the exemplary process 130, in a first iteration of stage 132, the user (e.g., an engineer) considers / theorizes appropriate modifications to the inspection algorithm / recipe, and / or modifications to the hardware settings (e.g., different cameras, lenses, lighting device types, etc., and / or different arrangements and / or settings of such devices / components) to attempt to correct the problem observed in stage 104. Thereafter, in a first iteration of stage 134, the operator modifies the hardware and / or code based on the modifications identified in the first iteration of stage 132.

[0032] In a first iteration of stage 136, the simulated AVI station is used to test whether its performance is satisfactory, that is, whether the problem observed in stage 104 has been corrected to a sufficient extent. For example, stage 136 may involve comparing statistical results (e.g., false positive rate, etc.) with standard-based requirements. Each iteration of stage 136 may be time-intensive and / or labor-intensive because it may require a large number of images and / or a variety of container / product samples to determine whether the problem has been resolved (e.g., if the observed problem is a low but still acceptable false positive rate or false negative rate). However, the time investment may be acceptable because it does not require interruption of commercial production line equipment.

[0033] If the performance is not determined to be satisfactory / acceptable in stage 136, process 130 is repeated, where new modifications are identified / theorized in a second iteration of stage 132. Process 130 can be repeated for any number of iterations without interrupting the operation of the production line equipment until the performance is determined to be satisfactory / acceptable in an iteration of stage 136. At this point, the troubleshooting process 130 is complete, and if the qualification and commissioning activities are successfully performed (at stage 140), the modifications made during process 130 (i.e., as reflected in the final state of the simulated AVI system after the final iteration of stage 134) are applied to the problematic AVI station in stage 142. While stage 142 typically requires stopping production of the commercial production line equipment in order to make the changes from process 130 (and possibly some brief qualification / commissioning operations), the downtime is significantly shorter than if the troubleshooting process 130 had to be performed on-site at the problematic AVI station itself instead. After the changes are applied to the problematic AVI station in stage 142, production (i.e., normal / production operation of the production line equipment) is resumed in stage 144.

[0034] While process 100 has been described with reference to troubleshooting a problematic AVI station, it should be understood that process 100 may alternatively be used to improve an already performing AVI station (e.g., to further optimize the inspection performance of the AVI station or to make the AVI station more cost-effective, etc.). Furthermore, while process 100 has been described with reference to the fill-finish phase of a pharmaceutical production line, it should be understood that process 100 may alternatively be used at a different phase (e.g., when inspecting a product after device assembly, or when inspecting the labeling and / or packaging of a product, etc.), and / or may alternatively be used in a non-pharmaceutical context (e.g., another context with relatively stringent quality standards).

[0035] Figure 1A and Figure 1B Depicted is a process 100 for troubleshooting an AVI station for commercial production line equipment by building and using a simulated AVI station. Figure 2 An example process 200 is depicted in which an AVI station on a commercial production line device is upgraded to simulate the performance of a laboratory-based AVI setup. For example, Figure 2 Commercial production line equipment includes one or more AVI stations and can be combined with the above Figure 1A and Figure 1B Or refer to the following Figure 3 Any type of production line equipment as discussed (ie, commercial production line equipment 302).

[0036] At stage 202, the commercial production line equipment is running in a normal / production mode of operation, for example, to perform a fill completion stage inspection of a particular drug product (e.g., a syringe filled with a drug). Figure 1A and Figure 1B As discussed, Figure 2 The process 200 may alternatively be applied to different inspection stages (eg, device assembly, packaging, etc.), and / or the process 200 may be used in non-pharmaceutical situations. Figure 2 The top horizontal line / arrow in FIG. 1 extending from stage 202 to stage 220 (discussed below) represents uninterrupted production line inspection using commercial production line equipment. Figure 2 The horizontal axis of the graph generally represents time, but is not necessarily drawn to scale, and Figure 2 The order of operations is not necessarily (but may be) indicated (eg, stage 202 may begin before or after stage 204 begins).

[0037] At stage 204, a decision is made to adapt the commercial production line equipment for fill completion inspection of the new drug product. For various reasons, the new product may require custom modifications to one or more AVI stations of the commercial production line equipment. For example, the new drug product may be less transparent than the previous product (e.g., requiring greater light intensity for imaging), or may be placed in a different type of container with different types and / or areas of potential defects, etc.

[0038] Next, a laboratory-based setup is used to develop an AVI station customized for the new drug product in a development process 210. Within the development process 210, at stage 212, the imaging system (one or more cameras and any associated optical components) of the laboratory-based setup captures an image of an illuminated sample (e.g., a fluid or lyophilized product) in a container (e.g., a syringe or vial).

[0039] At stage 214, with the help of the captured images, the user of the laboratory-based settings develops an inspection recipe / algorithm and tunes various parameters of the laboratory-based settings in an effort to achieve the desired performance (e.g., false positives and / or false negatives for (multiple) specific types of defects are less than a threshold amount). The tuned "parameters" may include any settings, types, locations, and / or other characteristics of the software, imaging hardware, lighting hardware, and / or computer hardware. For example, the user may adjust light intensity settings, camera settings, camera lens type or other optical components, the geometry of the imaging and illumination system, etc. It should be understood that the term "user" as used throughout this disclosure may refer to a single person or a team of two or more individuals. In some scenarios, the user may develop an entirely new software algorithm at stage 214.

[0040] In stage 216, the user performs characterization and qualification work to determine whether the laboratory-based setup / station, as developed / tuned in stage 214, performs in a satisfactory manner (e.g., in accordance with applicable regulations). If not, further development / tuning can be performed using the laboratory-based setup / station in another iteration of stage 214, which may also require capturing additional images in another iteration of stage 212. Stages 212, 214, 216 of the development procedure 210 can be repeated for any number of iterations until the results of the characterization / qualification work in stage 216 are deemed satisfactory.

[0041] When the results are deemed satisfactory and the new product is ready for commercial scale production, the commercial production line equipment is shut down and the hardware and / or software of the AVI stations of the production line equipment are updated at stage 220 to simulate the performance of the laboratory based setup. Figure 2 Explicitly shown in FIG, but the simulation process in stage 220 may involve Figure 1AThe iterative setup procedure 120 (i.e., stages 122, 124, 126) is similar to the procedure described above, but due to the fact that there is already an existing AVI station being updated, the initial iteration of stage 122 (potentially) does not require any "build." In fact, in some scenarios, the first iteration of stage 122 can be skipped entirely (i.e., if the developer believes that the existing AVI station is already close enough to the lab-based setup to proceed directly to the image comparison tool stage) and only performed for subsequent iterations.

[0042] It may also be necessary for the developer to perform some level of on-site qualification work at stage 220 (after a successful update based on the image comparison tool), which may increase production line equipment downtime. However, this scenario may require significantly less time than the qualification work during the development process 210, and in any case, production line equipment downtime is significantly reduced due to the fact that the development work (at process 210) occurs offline. At stage 222, after successful qualification, the commercial production line equipment resumes production, but now producing the new product.

[0043] In some scenarios, both process 100 and process 200 are implemented sequentially. For example, if it is determined that an AVI station of production line equipment is generating an unacceptable number of false positives (i.e., flagging defects where no significant defects exist), process 100 can be used to build and tune a simulated AVI station. Thereafter, by using the simulated AVI station, it can be determined that different hardware components are required (e.g., a single LED ring light instead of multiple directional lights). After a new design for the simulated AVI station has been identified, process 200 can be used when upgrading the AVI station in the production line equipment to ensure that the upgraded station accurately / adequately matches the performance of the simulated station.

[0044] Figure 3 is a simplified block diagram of an example system 300 that can implement the techniques described herein. Specifically, Figure 3 An embodiment of using a simulated AVI station to troubleshoot an AVI station of commercial production line equipment is depicted. Thus, for example, the system 300 may be implemented and / or used to implement Figure 1A and Figure 1B process 100.

[0045] like Figure 3 As shown, the system 300 includes a commercial production line device 302 and a simulated AVI station 304. The production line device 302 can be any production-level device having N (N≥1) AVI stations 310-1 to 310-N (also collectively referred to as AVI stations 310). To provide only one example, the production line device 302 can be 296S production line equipment. Figure 3In the example of FIG, the i-th AVI station 310-i of the production line equipment 302 needs to be troubleshooted (or alternatively, optimized), where i is equal to 1, N, or any number between 1 and N. Each of the AVI stations 310 can be responsible for capturing images to be used to inspect different aspects of the container and / or the sample within the container. For example, the first AVI station 310-1 can capture an image of a top view of a syringe or vial to inspect for cracks or chips, and the second AVI station 310-2 can capture a side view image to inspect the contents of the syringe or vial (e.g., a fluid or a lyophilized drug product) for foreign particles, etc.

[0046] Figure 3 The general components of the i-th AVI station 310-i are shown in simplified block diagram form. In particular, the AVI station 310-i includes an imaging system 312, an illumination system 314, and sample positioning hardware 316. It should be understood that other AVI stations 310 (if any) may be similar, but may have different component types and configurations, which are appropriate for the purpose of each given station 310.

[0047] The imaging system 312 includes one or more imaging devices and may include associated optical components (e.g., additional lenses, mirrors, filters, etc.) to capture images of each sample (e.g., container and pharmaceutical product). For example, these imaging devices can be cameras with charge coupled device (CCD) sensors. As used herein, the terms "camera" or "imaging device" can refer to any suitable type of imaging device (e.g., a camera that captures a portion of the spectrum visible to the human eye, or an infrared camera, etc.). For example, the illumination system 314 includes one or more illumination devices, such as a light emitting diode (LED) array (e.g., a panel device or a ring device), for illuminating each sample for imaging.

[0048] The sample positioning hardware 316 may include any hardware that holds or otherwise supports the sample, and may include hardware that transports and / or otherwise moves the sample for the AVI station 310-i. For example, the sample positioning hardware 316 may include a star wheel, a carousel, a robotic arm, etc. In some embodiments, depending on the function of the AVI station 310-i, the sample positioning hardware 316 also includes hardware for agitating each sample. For example, if the AVI station 310-i inspects a liquid for foreign particles, the sample positioning hardware 316 may include components for turning / rotating, inverting, and / or shaking each sample.

[0049] Production line equipment 302 also includes a processing unit 320 and a memory unit 322. Processing unit 320 may include one or more processors, each of which may be a programmable microprocessor that executes software instructions stored in memory unit 322 to perform some or all of the software-controlled functions of production line equipment 302 as described herein. Alternatively or additionally, some processors in processing unit 320 may be other types of processors (e.g., application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc.), and certain functions of processing unit 320 as described herein may alternatively be implemented in hardware. Memory unit 322 may include one or more volatile and / or non-volatile memories. Memory unit 322 may include any suitable memory type or types, such as read-only memory (ROM), random access memory (RAM), flash memory, solid-state drive (SSD), hard disk drive (HDD), etc. Memory unit 322 may collectively store one or more software applications, data received / used by these applications, and data output / generated by these applications.

[0050] The processing unit 320 and the memory unit 322 are collectively configured to control / automate the operation of the AVI station 310 and process images captured / generated by the AVI station 310 to detect whether a container and / or container contents (e.g., a pharmaceutical product) have corresponding types of defects. In an alternative embodiment, the functions of the processing unit 320 and / or the memory unit 322 are distributed among N different processing units and / or memory units, each of which is specific to a different AVI station among the AVI stations 310-1 to 310-N. In another embodiment, certain functions of the processing unit 320 and the memory unit 322 (e.g., for transporting, shaking, and / or imaging samples) are distributed among the AVI stations 310, while other functions of the processing unit 320 and the memory unit 322 (e.g., for processing sample images to detect defects) are performed by a centralized processing unit. In some embodiments, at least a portion of processing unit 320 and / or memory unit 322 is included in a computing system (eg, a specially programmed general-purpose computer) that is external to (and possibly remote from) production line equipment 302 .

[0051] Memory unit 322 stores sample (container / product) images captured by AVI station 310 and also stores AVI code 326 that, when executed by processing unit 320, performs two operations: (1) causes AVI station 310 to capture an image; and (2) processes the captured image to detect defects (e.g., as discussed above). For AVI station 310-i, for example, AVI code 326 includes Figure 3328. As an example of one embodiment, code 328 may trigger imaging system 312 to capture images while the sample is being illuminated by illumination system 314, and may control sample positioning hardware 316 to place the sample in the correct position at the appropriate time, and may also shake the sample according to a shaking curve at the appropriate time. After capturing the images and storing them as images 324, code 328 processes images 324 to detect a specific type of defect associated with station 310-i. As described above, in some embodiments, the portion of code 328 that processes the images may be executed by a different processor, component, and / or device than the portion of code 328 that controls imaging, shaking, etc.

[0052] The simulated AVI station 304 can be a laboratory-based setup that is constructed to attempt to replicate (to a sufficient degree) the performance of a particular AVI station 310-i (e.g., in response to learning that the AVI station 310-i has an unacceptable level of false positives or false negatives). The simulated AVI station 304 includes a simulated imaging system 332, a simulated illumination system 334, and simulated sample positioning hardware 336. The simulated imaging system 332 includes one or more imaging devices (and possibly associated optical components) for capturing an image of each sample (e.g., a container and a pharmaceutical product), the simulated illumination system 334 includes one or more lighting devices for illuminating each sample for imaging, and the sample positioning hardware 316 includes hardware that holds or otherwise supports the sample and may include hardware that transports and / or otherwise moves the sample.

[0053] Ideally, the simulated imaging system 332, the simulated illumination system 334, and the simulated sample positioning hardware 336 would perfectly replicate the imaging system 312, the illumination system 314, and the sample positioning hardware 316 of the AVI station 310-i, respectively. More importantly, the simulated AVI station 304 as a whole would perfectly replicate the performance of the AVI station 310-i. However, in the real world, exact matching of performance is very difficult to achieve. As discussed above in conjunction with Figure 1A As described, various manual and / or automated reverse engineering techniques may be used to construct a simulated AVI station 304 (eg, at stage 122) that initially has performance "close to" that of the AVI station 310-i.

[0054] After initially constructing the simulated AVI station 104, as described above with reference to Figure 1AAs discussed in stage 126 of the present invention, software can be used to facilitate fine-tuning of the simulated AVI station 304 to improve performance matching. To this end, the simulated AVI station 304 is coupled to a computing system 340 (e.g., a specially programmed general-purpose computer) that includes a processing unit 342 and a memory unit 344. The computing system 340 can be separate from or integral to the simulated AVI station 304 and located near or remote from the simulated AVI station 304. In some embodiments, for example, the computing system (or a portion thereof) receives images from the simulated AVI station 304 via an Internet link.

[0055] The processing unit 342 may include one or more processors, each of which may be a programmable microprocessor that executes software instructions stored in the memory unit 344 to perform some or all of the software-controlled functions of the computing system 340 as described herein. Alternatively or additionally, some of the processors in the processing unit 342 may be other types of processors (e.g., ASICs, FPGAs, etc.), and certain functions of the processing unit 342 as described herein may alternatively be implemented in hardware. The memory unit 344 may include one or more volatile and / or non-volatile memories. The memory unit 344 may include any suitable memory type or types, such as ROM, RAM, flash memory, SSD, HDD, etc. The memory unit 344 may collectively store one or more software applications, data received / used by these applications, and data output / generated by these applications.

[0056] Memory unit 344 stores images 346 captured by analog imaging system 332 and images 348 captured by imaging system 312 of AVI station 310-i. Memory unit 344 also stores an image comparison tool (ICT) 350 and AVI code 352. Generally, image comparison tool 350 facilitates the process of tuning analog AVI station 304 so that its performance matches that of AVI station 310-i (e.g., as described above in connection with stage 126 and below in connection with Figure 6 and Figure 7 328 ), and the AVI code 352 is used to control the constructed and tuned simulated AVI station 304 during the troubleshooting or optimization process. For example, the AVI code 352 can be a perfect (or very close) copy of the code 328 and can be downloaded or uploaded from the production line equipment 302, from a portable memory device, from the Internet, or from another suitable source. In other embodiments, the AVI code 352 includes only a portion of the code 328 (e.g., does not include the portion used to control the transport of samples to and from the appropriate imaging location).

[0057] Computing system 340 is coupled to output unit 360, which can be any type of visual and / or audio output device (e.g., a computer monitor, a touch screen or other display, and / or speakers of computing system 340, or a separate computing device having a display and / or speakers and coupled to computing system 340, etc.). Image comparison tool 350 and AVI code 352 can cause output unit 360 to provide various visual and / or audio outputs to a developer or user of simulated AVI station 304. For example, image comparison tool 350 can cause output unit 360 to display various indicators representing differences between image 346 and image 348 (as discussed further below), and AVI code 352 can cause output unit 360 to display information such as an indicator of whether a particular sample is defective.

[0058] Although Figure 3 An embodiment is depicted in which a simulated AVI station (station 304) is used to troubleshoot an AVI station of commercial production line equipment (station 310-i of production line equipment 302), but it should be understood that similar components can be used in any application where the AVI station of commercial production line equipment is updated / modified to replicate the performance of a laboratory-based setup / station (e.g., as in Figure 2 In one embodiment, the image comparison tool 350 may be located in the memory 322 of the production line equipment 302. Alternatively, the image comparison tool 350 may be located in the computing system 340.

[0059] In some embodiments, system 300 provides access to remote sites (e.g., global manufacturing sites) to enable remote users to directly access laboratory-based settings, regardless of the respective laboratory and manufacturing locations. Such an approach enables real-time collaboration between sites / users across the network and enables troubleshooting and / or development support to be located at a centralized global facility, for example, while still leveraging the expertise of individuals (engineers, etc.) located in various locations. Thus, a networked approach can lead to a more efficient organizational structure.

[0060] Figure 4 and Figure 5 Depicts example lab-based settings, any of which can be Figure 3 The simulated AVI station 304 of process 100 may be a simulated AVI station (e.g., the simulated AVI station of process 100) or alternatively a lab-based AVI station used as a development platform (e.g., the lab-based setup used in process 200). It should be understood that these lab-based setups / stations are merely illustrative and that there are an almost infinite number of alternative types and configurations.

[0061] First reference Figure 4A For example, a first laboratory-based setup 400 can be used to inspect the tops of containers (e.g., the tops of syringes filled with liquid drug product) for defects (e.g., cracks, chips, etc.). Setup 400 includes a camera 402 for imaging the containers and an LED ring 404 for illuminating each container while imaging the container. Sample positioning hardware 406 includes a platform 406A and a star wheel 406B mounted on platform 406A. Camera 402 and LED ring 404 can also be mounted (directly or indirectly) on platform 406A. Star wheel 406B can hold containers (e.g., syringes) in fittings along the perimeter of star wheel 406B, and star wheel 406B rotates relative to platform 406A to move each container into imaging position (i.e., centered within LED ring 404 and directly below camera 402). However, in some embodiments and / or scenarios, it is only important that the setup 400 matches or closely approximates the characteristics (e.g., lighting, optics, etc.) of the production line equipment for imaging a single container, rather than being able to sequentially image multiple consecutive containers. Thus, in some embodiments, the star wheel 406B does not need to be designed to hold multiple containers or rotate, even if the AVI station being simulated (or the AVI station undergoing development activities) requires such hardware or functionality.

[0062] Figure 4B Example images 450 that may be captured by setup 400 are depicted for an embodiment in which setup 400 (and a corresponding AVI station of production line equipment) captures top-view images of syringes to detect defects on syringe flanges. As shown in this example, these defects may include various locations and sizes of nicks and cracks on the syringe flange.

[0063] Figure 4C Depicts an AVI station representing a production line device (e.g. Figure 3 Example image 460 of an image captured by the AVI station 310-i) and Figure 4D An example image 470 is depicted representing an image captured by a simulated AVI station (e.g., setup 400) after successful troubleshooting. In this example, the production line equipment AVI station may have an unacceptably high false positive rate due to light reflections as shown in image 460, and modifications during troubleshooting (e.g., changing from another type of lighting fixture to an LED ring, or changing the orientation of the camera or lighting fixture(s) relative to the container, etc.) resulted in a significant reduction in reflections, as shown in image 470. The reduction in artifacts associated with light reflections can reduce the likelihood of defects being obscured and / or reduce the likelihood of reflections being erroneously interpreted as defects.

[0064] Now refer to Figure 5For example, a laboratory-based setup 500 can be used to inspect syringes for defects (e.g., cracks, nicks, stains, etc. in the sidewall of the container syringe, and / or unacceptable types and / or quantities of particles within the drug product) from a side view (e.g., the side of a syringe filled with a liquid drug product). Setup 500 includes a camera 502 for imaging the containers, and an LED backlight 504 for illuminating each container from behind while imaging the container. Sample positioning hardware includes a carousel 506 that holds multiple syringes and positions a single syringe between camera 502 and LED backlight 504. Carousel 506 can rotate to move each syringe into imaging position. As described above, in some embodiments and / or scenarios, it is only important that setup 500 matches or closely approximates the characteristics of an AVI station of a production line equipment in terms of imaging a given sample rather than having the same ability to sequentially image multiple consecutive samples. Thus, in some embodiments, the carousel 506 need not be designed to hold multiple syringes or to rotate, even if the AVI station being simulated (or the AVI station undergoing development activities) requires such hardware or functionality.

[0065] Figure 6 Depicts a lab-based setup that can be used to facilitate a simulated AVI station (e.g., in process 100 of FIG. 1 , Figure 2 AVI station in the process 200 production line equipment or Figure 3 Example image comparison tool 600 (e.g., Figure 3 Image comparison tool 350). Figure 6 As shown, image comparison tool 600 receives an original image 602 and a simulated image 604, where original image 602 may be an image captured by an AVI station of production line equipment and simulated image 604 may be an image captured by a laboratory-based setting, or vice versa, depending on whether process 100 or process 200 is being implemented. In some embodiments, images 602, 604 are assumed to use a lossless image compression format.

[0066] Image comparison tool 600 includes a metric generation unit 612 and a feedback unit 614. Metric generation unit 612 processes images 602 and 604 and generates / calculates metrics indicating characteristics of images 602 and 604. Based on these metrics, it calculates one or more additional metrics indicating differences between images 602 and 604. In some embodiments, metric generation unit 612 calculates the metric(s) on an image-by-image basis, i.e., by comparing a single original image in original images 602 with a single simulated image in simulated images 604. In other embodiments, metric generation unit 612 calculates each of the metric(s) based on a collection of multiple original images 602 and multiple simulated images 604. For example, after applying an alignment technique, a set of x images (x > 1) of original images 602 and a set of x images of simulated images 604 may be averaged or superimposed on one another, and a set of x images of simulated images 604 may be averaged or superimposed on one another. Such an approach can, for example, reduce the impact of outlier images. The averaging / superimposition / alignment can be performed, for example, by the same computing device or processor that implements image comparison tool 600, or by another device or processor. In alternative embodiments (e.g., if a line scan camera is used), the x images of the original images 602 are stitched together and the x images of the simulated images 604 are stitched together before unit 612 calculates any metrics (e.g., in embodiments where each container is rotated to obtain a 360 degree view of the container). Other pre-processing of the images 602 and 604 is also possible (e.g., generating an image across all x images where each pixel has the maximum intensity at that pixel location, etc.). For ease of explanation, Figure 6 The rest of the description and Figure 7 The description of only mentions one original image 602 and one simulated image 604. However, it should be understood that the above variations or other suitable variations are possible. Figure 7 Some specific examples of metrics that the metric generation unit 612 may calculate are discussed.

[0067] Based on the difference metric(s) generated by the metric generation unit 612, the feedback unit 614 causes one or more outputs to be presented to a user (e.g., an engineer, developer, technician, etc.) in a visual and / or audio format. For example, these outputs may be Figure 3 The output unit 360 generates (e.g., displays and / or issues) the generated indicator(s) to the user (e.g., on a graphical user interface (GUI)). For example, the feedback unit 614 may display an intensity level indicator, an image defocus indicator, and / or other indicators to the user.

[0068] In other embodiments, the feedback unit 614 generates one or more user suggestions based on one or more of the indicators, alternatively or additionally. For example, the feedback unit 614 can be used to generate suggestions to move the camera, move the lighting device, change the light intensity, change the camera settings, etc. Figure 7 Discuss other sample suggestions.

[0069] In some embodiments, the metrics generation unit 612 and the feedback unit 614 operate substantially in real time as the simulated image 604 is captured by the imaging system of the simulated AVI station (e.g., by the simulated imaging system 332) and / or as the simulated image 604 is received by a device or system implementing the image comparison tool 600 (e.g., by the computing system 340). Thus, for example, a user may be able to capture a simulated station image by selecting an interactive control on a GUI presented on the output unit 360, and then almost immediately view the corresponding metrics and / or suggestions generated by the metrics generation unit 312 and / or the feedback unit 314. In this way, the user can quickly go through iterations of modifying the simulated AVI station (e.g., slightly adjusting component positions, settings, etc.) and observing the effects of the modifications on the performance of the simulated AVI station (i.e., how the modifications may cause the simulated AVI station to move closer to or further away from the performance of the original AVI station).

[0070] For example, Figure 7 Describes the Figure 6 Example algorithm 700 implemented by the image comparison tool 600 (and more specifically, by the indicator generation unit 612). Figure 7 As shown, algorithm 700 accepts as input a single original image 702 (e.g., one of images 602) and a single simulated image 704 (e.g., one of images 704). As described above, in some embodiments, these "single" images may be a composite (e.g., an average) of multiple images. In one embodiment, algorithm 700 is implemented in Python code using the OpenCV library.

[0071] At stage 712, the algorithm 700 determines one or more overall image parameters (P1 to Pj, where j ≥ 1) of the original image 702 and the simulated image 704 and compares these parameters to check for consistency. For example, these overall image parameters can represent relatively basic image parameters such as image size, image resolution, color depth, and / or image file format. For example, the image comparison tool 600 can determine the overall image parameters using various hardware components (e.g., cameras) input to the original and simulated AVI stations or configuration files generated by these hardware components. The algorithm 700 can enable a GUI (e.g., a Figure 3Any differences in the overall image parameters are indicated by the output unit 360 display of the simulating AVI station. Thus, for example, a user may be able to easily detect when the camera (or subsequent processing) in the original AVI station has altered the captured image in some way (e.g., cropped, format converted, resized, etc.), and may manually reconfigure the simulated AVI station to reproduce those image processing operations.

[0072] At stages 714A and 714B, the algorithm 700 calculates one or more metrics (C1 to Ck, where k ≥ 1) for the original image 702 and the simulated image 704, respectively. Stages 714A and 714B may assume that the overall image parameters of the images 702, 704 are the same. These metrics may represent any image characteristics that are or may be relevant to the accuracy of the inspection. The algorithm 700 also includes a metric (in the example of the example of the algorithm 700) indicating the difference between the corresponding metrics of the two images 702, 704. Figure 7 The algorithm may compute some or all of the differences, for example, by simple subtraction (e.g., where ΔC1 is equal to the absolute value of the difference between C1 of image 702 and C1 of image 704, etc.) or using other techniques (e.g., element-wise subtraction, dot products, etc.).

[0073] As an example, these indicators may include one or more light intensity indicators. Depending on the embodiment, light intensity may be measured in one or more ways. For example, for each of images 702, 704, the algorithm 700 may average the intensity values of all pixels to generate an average value, and compare (e.g., perform a subtraction) the average values of images 702, 704. As another example, the algorithm 700 may generate a pixel intensity histogram for each of images 702, 704, and then use known techniques to mathematically compare these histograms. Some techniques that may be used and provide a single value output reflecting the differences between the histograms include the Bhattacharya distance method, the correlation method, the chi-square method, and the intersection method. The best technique to use may depend on the nature of the image under consideration. Histogram-based intensity analysis allows the algorithm 700 to take into account the dynamic intensity range (i.e., the spread between the minimum and maximum intensity values) within each image.

[0074] For complex images, intensity metrics (such as those discussed above) may not be sufficient because non-uniform illumination may be disproportionately reflected from a subset of the container's facets and its immediate surroundings. In addition, the roughness effect caused by non-uniform ambient illumination (e.g., from a window or fluorescent ceiling light) may result in large wavelength, gradual intensity variations across the image. In some embodiments, the algorithm 700 uses a low-pass frequency filter to capture such variations. Additionally or alternatively, the algorithm 700 may calculate a fast Fourier transform (FFT) for each of the images 702, 704 and process the corresponding FFT outputs to determine whether the frequency content is similarly distributed for the images 702, 704.

[0075] As another example, these indicators may include one or more indicators indicating image defocus. For example, algorithm 700 may calculate the Laplacian operator for each of images 702, 704 to generate a single-valued parameter that can be easily compared. For example, if algorithm 700 is implemented as Python code using the OpenCV library, defocus may be calculated for each of images 702, 704 as:

[0076] Defocus=cv2.Laplacian(image,cv2.CV_64F).var()

[0077] The Laplace operator metric can also indicate the motion of the imaged container / sample relative to the imaging camera(s). In commercial AVI systems, it is common to move the sample rapidly relative to the inspection station. To achieve clear imaging, for example, commercial systems will typically employ short camera exposure times, strobing of the illumination lamp, motion of the imaging components used to track the part, or a combination of one or more of these features. If one or more of these features of the simulated AVI station do not match that of the original AVI station, a degree of motion blur or image smearing may occur. The result is similar to defocus, but with a directional component. Therefore, the metrics calculated using the Laplace operator technique can also indicate motion. In addition to the Laplace operator, first-order filters such as Sobel or Prewitt can be used to obtain additional information about image clarity.

[0078] As another example, these metrics may include one or more indicators of camera noise. For modern digital industrial cameras, noise can only occur in the sensor itself. Once the signal is digitized, it is generally no longer susceptible to corruption. Because camera brands and models can in some cases be easily matched when setting up an analog AVI station, noise levels can be similar. As an example, for a particular camera brand and model, the cumulative noise level at a given pixel in an 8-bit grayscale image can be in the range of 0 to 10 within a dynamic range of 0 to 255. This makes the noise at room temperature negligible for most AVI applications. However, it is conceivable that in some cases, camera noise levels may affect inspection performance and / or the camera model may be outdated and unavailable. For digital images, camera noise is typically high-frequency in nature. Therefore, algorithm 700 can generate an indicator indicating camera noise by calculating an FFT for each of images 702, 704 and processing the FFT output to generate an indicator indicating the level or relative level of high-frequency components.

[0079] As another example, these metrics may include one or more metrics indicative of the alignment and scaling of the imaged container (and possibly other imaged objects, such as a portion of the sample positioning hardware). For example, the algorithm 700 may determine the relative rotation of the imaged container (e.g., the angle of a vertical wall of the imaged container relative to the vertical or horizontal axis of the image itself (i.e., relative to the axis established by the camera frame)) and the lateral and / or proportional depth offset / shift (e.g., determined based on the length and / or width as measured in pixels).

[0080] At stage 716, the algorithm 700 generates a weighted comparison score for the image pair 702, 704 based on the metrics calculated at stages 714A and 714B. In particular, in this example, the algorithm 700 calculates the score as SCORE = (W1*ΔC1) + (W2*ΔC2) + ... + (Wk*ΔCk), where (as described above) ΔCi is the difference indicator corresponding to Ci of the two images 702, 704. The weights W1, W2, ... Wk can represent the importance of achieving similarity for each metric, specifically with respect to which metrics are more or less important for achieving equivalent AVI inspection performance. These weights may be application-specific to some extent. In some embodiments or scenarios, for example, pixel intensity levels may be more important than defocus, and thus the intensity difference may be weighted more heavily than the difference in Laplacian operator scalar values (or other metrics indicative of defocus).

[0081] As above combined Figure 6As described above, the image comparison tool 600 can cause the output unit 360 to display some or all of the calculated metrics (e.g., only the difference metrics ΔC1 to ΔCk, or only the difference metrics that exceed corresponding thresholds, etc.) in real time. Alternatively or additionally, the tool 600 can cause the output unit 360 to display the score generated at stage 716. In embodiments where the score is calculated and displayed and is deemed sufficient (e.g., above a predetermined threshold), the engineer or other user can determine that the simulated AVI station adequately matches the original AVI station and continue to use the simulated AVI station for troubleshooting and / or optimization.

[0082] If the score is insufficient, or if no score is presented and the individual metrics do not appear close enough, the user can analyze the displayed metrics to "slightly adjust" the simulated AVI station as needed to achieve a better match. For example, if the metrics show that the simulated image 704 as a whole is dim (e.g., has a low average intensity) relative to the original image 702, the user might check the lighting or camera device settings (e.g., light intensity setting, lens aperture setting, camera gain setting, camera exposure time setting, etc.), and / or move the lighting device closer to the sample, etc.

[0083] There are multiple aspects associated with the illumination source that may affect the intensity of the image pixels. Typically, in typical AVI applications, the impact may be macroscopic in the entire image. High-end industrial LEDs are typically used in modern AVI stations. When setting up an analog AVI station, if the LED source is not placed at the correct distance from the container, the intensity may drop. Intensity drops as the square of the distance from the source, and therefore a moderate difference in positioning may have a detectable effect on the final image. Once its position relative to the container is set, the brightness of the LED source can still vary due to power supply. Once all other factors have been eliminated, the image comparison tool 600 can be used in real time to fine-tune the power and subsequent brightness of the LED source. Image processing technology (such as the image processing technology above) provides a more accurate and detailed solution than the conventional method of using an illuminometer to measure the brightness of a light source in AVI applications.

[0084] Some optical components, such as lenses, may include a manual aperture or other feature that can also affect overall image intensity (and possibly other image characteristics, such as the aperture setting that affects image sharpness). These are typically manual dials or screws on the lens, with no digital feedback. In some cases, these components also lack any kind of visible gradient or scale. Telecentric lenses are often used in factory automation inspection stations to achieve high-fidelity images of the product. These lenses contain a pinhole aperture, and in some models the size of this pinhole aperture can be manually adjusted. This has an impact on the amount of light allowed through the lens, and has a specific effect on the sharpness of the image. Therefore, by considering and combining both factors, the user observing the indicator can appropriately set the lens characteristics related to intensity.

[0085] Large, uniform intensity drops across the image can also indicate incorrect filter placement across the illumination source (such as a polarizer or diffuser) or in front of the camera (such as an alternative polarizer or wavelength filter). Therefore, when large differences in image intensity are observed, the user can try adjusting the filter placement.

[0086] As another example, a user examining the intensity histograms of images 702, 704 and / or the output of low-pass frequency filtering can determine whether there are significant local differences in intensity between images 702, 704. If local differences do exist, the user can attempt to identify and remove the sources of these local differences by studying the images 702, 704 and the histograms and / or other metrics.

[0087] As another example, if the indicator indicates that the container in the simulated image 704 is misaligned and / or inappropriately scaled relative to the container in the original image 702, the user may adjust the alignment of the container and / or camera (e.g., angle or rotation), the distance between the container and the camera, the zoom level of the camera (e.g., lens type or digital zoom setting), etc.

[0088] As another example, if an indicator (e.g., a difference in the scalar Laplacian output) indicates that one of images 702, 704 is defocused relative to the other, the user can adjust the distance between the container and the camera, the motor speed, the camera exposure time, etc. Telecentric lenses of the type often used in inspection applications typically have a very short depth of field, so that small errors in the relative placement of the lens and the container can result in a blurred image. Therefore, even small distance differences can have a significant effect on defocus. Furthermore, as discussed above, some of these lenses have a variable pinhole aperture, which can also cause image blur if improperly set. Therefore, if the indicator indicates a defocus difference, the user can also adjust the aperture.

[0089] Other metrics can guide the user in adjusting these and / or other aspects of the simulated AVI station, for example, to achieve a better match of stray reflections in the image, presence of key objects in the image, image dynamic range, image bleaching, image contrast, etc.

[0090] In some embodiments, as described above, the image comparison tool 600 generates one or more recommendations based on these indicators. Thus, for example, the image comparison tool 600 can cause the output unit 360 to display (and / or generate a computer voice message describing) any of the remedial techniques discussed above (e.g., if the indicator reflecting the intensity and / or defocus difference between the images 702, 704 is above a threshold level, then increase or decrease the distance between the camera and the container, and possibly increase or decrease the lens aperture, etc.).

[0091] Figure 8 8 is a flow chart of an example method 800 for replicating the performance of an AVI station (e.g., AVI station 310-i in a troubleshooting scenario, or a lab-based setting for development, etc.). In method 800, at block 802, a simulated AVI station (e.g., simulated AVI station 304 in a troubleshooting scenario, or AVI station 310-i, etc.) is constructed that performs one or more AVI functions of the original AVI station. Block 802 can be performed manually by reverse engineering the original AVI station. However, in some embodiments, block 802 includes using software to assist in the reverse engineering process (e.g., by interpreting a CAD file or the output of a 3D scanner). For example, block 802 can include the first iteration of stage 122.

[0092] At block 804, one or more container images (i.e., images of the container at appropriate imaging locations within the original AVI station) are captured by the imaging system of the original AVI station (e.g., a single camera). At block 806, one or more additional container images are captured by the imaging system of the simulated AVI station (e.g., a single camera of the same type). For example, blocks 804 and 806 may be similar to stages 112 and 124, respectively, of process 100.

[0093] Thereafter, at block 808, one or more differences are identified between the container image(s) captured by the original AVI station and the container image(s) captured by the simulated AVI station. Block 808 may be performed by a processing unit (e.g., processing unit 342) executing an image comparison tool (e.g., tool 350). For example, the image comparison tool may generate one or more indicators reflecting the differences at block 808. For example, block 808 may include stages 714A and 714B of algorithm 700, as well as a difference indicator (e.g., Figure 7 Subsequent generation of indicators ΔC1 to ΔCk) in .

[0094] At block 810, a visual indication of the difference(s) identified at block 808 is generated to assist the user in modifying the simulated AVI station. The visual indication may include, for example, one or more of the metrics (e.g., the difference metric) calculated at block 808 and / or one or more suggestions based on these metrics (e.g., as described above in conjunction with Figure 6 and Figure 7 For example, block 810 may be performed by the same processing unit that performed block 808 and may include causing an output unit (e.g., output unit 360) to present a visual indication (i.e., indicator(s) and / or suggestion).

[0095] At block 812, the simulated AVI station is modified based on the visual indication generated at block 810. Block 812 may be performed entirely by a user (i.e., manually) or may be performed at least partially automatically (e.g., by the computing system 340 adjusting digital settings of a camera or lighting device of the simulated AVI station, etc.). For example, block 812 may comprise a second (or subsequent) iteration of stage 122 of process 100.

[0096] Although the systems, methods, devices, and components thereof have been described in terms of exemplary embodiments, they are not limited thereto. The detailed description is to be construed as exemplary only and does not describe every possible embodiment of the present invention, as describing every possible embodiment would be impractical, if not impossible. Numerous alternative embodiments may be implemented using current technology or technology developed after the filing date of this patent, which would still fall within the scope of the claims defining the present invention.

[0097] Those skilled in the art will recognize that various modifications, changes and combinations may be made to the embodiments described above without departing from the scope of the present invention, and such modifications, changes and combinations may be considered to be within the scope of the present invention.

Claims

1. A method for replicating the performance of an automated visual inspection (AVI) station, the method comprising: constructing a simulated AVI station that performs one or more AVI functions of the AVI station, wherein the simulated AVI station includes a simulated illumination system, a simulated imaging system, and simulated sample positioning hardware configured to hold and / or support a container containing a sample; capturing one or more images of the container by an imaging system of the AVI station while the container is illuminated by the illumination system of the AVI station; capturing one or more additional images of the container by the simulated imaging system while the container is illuminated by the simulated illumination system; identifying, by the one or more processors, one or more differences between the one or more additional container images and the one or more container images; generating, by the one or more processors, a visual indication of: (i) the one or more differences, and / or (ii) one or more suggestions for modifying the simulated AVI station; and The simulated AVI station is modified based on the visual indication by modifying at least the simulated illumination system, the simulated imaging system, and / or the simulated sample positioning hardware.

2. The method according to claim 1, wherein (i) the AVI station is included in commercial production line equipment and the simulated AVI station is a laboratory-based setting, or (ii) the simulated AVI station is included in the commercial production line equipment and the AVI station is the laboratory-based setting.

3. The method according to claim 1, wherein Modifications to this simulated AVI station include: modifying the spatial arrangement of at least one illumination device of the simulated illumination system, at least one imaging device of the simulated imaging system, and / or the simulated sample positioning hardware; and / or At least one illumination device of the simulated illumination system, at least one imaging device of the simulated imaging system, and / or hardware components of the simulated sample positioning hardware are modified.

4. The method of claim 2, further comprising: installing AVI software on the computer system of the simulated AVI station that implements an inspection algorithm also implemented by the commercial production line equipment; as well as After modifying the simulated AVI station, capturing one or more new container images by the simulated imaging system while a test container is illuminated by the simulated illumination system, and One or more characteristics of the test container and / or the sample within the test container are determined by processing the one or more new container images according to the inspection algorithm.

5. The method of claim 4, further comprising: Troubleshooting or optimizing the simulated AVI station based at least in part on the determined one or more characteristics, wherein troubleshooting or optimizing the simulated AVI station includes modifying the simulated illumination system, the simulated imaging system, the simulated sample positioning hardware and / or the AVI software, or iteratively making additional modifications to the modified simulated illumination system, the simulated imaging system, the simulated sample positioning hardware and / or the AVI software.

6. The method of claim 5, further comprising: The AVI station is modified according to the modifications or additional modifications made to the simulated illumination system, the simulated imaging system, the simulated sample positioning hardware and / or the AVI software.

7. The method of claim 1, wherein: The one or more container images include a first plurality of container images; The one or more additional container images include a second plurality of container images; and Identifying one or more differences between the one or more additional container images and the one or more container images includes identifying one or more differences between a first composite image derived from the first plurality of container images and a second composite image derived from the second plurality of container images.

8. The method of claim 1, wherein: Identifying one or more differences between the one or more additional container images and the one or more container images includes identifying differences with respect to: container alignment; Defocus; strength; intensity changes; motion blur; and / or Imaging sensor noise.

9. The method of claim 1, wherein: Identifying one or more differences between the one or more additional container images and the one or more container images includes computing (i) a fast Fourier transform (FFT) of the one or more container images or a composite image derived therefrom, and (ii) an FFT of the one or more additional container images or a composite image derived therefrom.

10. The method of claim 1, wherein: Identifying one or more differences between the one or more additional container images and the one or more container images includes generating (i) a histogram of pixel intensity levels of the one or more container images or composite images derived therefrom, and (ii) a histogram of pixel intensity levels of the one or more additional container images or composite images derived therefrom.

11. The method of claim 1, wherein: Identifying one or more differences between the one or more additional container images and the one or more container images includes computing (i) a Laplacian of the one or more container images or a composite image derived therefrom, and (ii) a Laplacian of the one or more additional container images or a composite image derived therefrom.

12. The method of claim 1, wherein: (i) identifying the one or more differences and (ii) generating the visual indication are performed in real time while capturing the one or more additional container images.

13. The method of claim 1, further comprising: generating, by the one or more processors, one or more suggestions for modifying the simulated AVI station based on the one or more differences, Wherein, generating the visual indication includes generating a visual indication of the one or more suggestions.

14. The method of claim 13, wherein: Generating the one or more suggestions includes generating: a recommendation for modifying one or more configurable settings of at least one imaging device of the simulated imaging system; a recommendation for modifying one or more configurable settings of at least one lighting device of the simulated illumination system; a suggestion for modifying a position of at least one imaging device of the simulated imaging system; and / or A suggestion for modifying a position of at least one lighting device of the simulated illumination system.

15. The method of claim 13, wherein: The simulated sample positioning hardware is configured to move the container according to the movement curve; and Generating the one or more suggestions includes generating suggestions for modifying one or more characteristics of the movement profile.

16. The method of claim 1, further comprising: receiving, by the one or more processors, one or more operating parameters of the imaging system and one or more operating parameters of the simulated imaging system; comparing, by the one or more processors, one or more operating parameters of the imaging system to one or more operating parameters of the simulated imaging system; as well as An additional visual indication of at least one operating parameter that differs between the imaging system and the simulated imaging system is generated by the one or more processors.

17. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: receiving one or more images of the container captured by an imaging system of an automated visual inspection (AVI) station; receiving one or more additional images of the vessel captured by the simulated imaging system of the simulated AVI station; identifying one or more differences between the one or more additional container images and the one or more container images; as well as A visual indication is generated of: (i) the one or more differences, and / or (ii) one or more suggestions for modifying the simulated AVI station.

18. The non-transitory computer readable medium of claim 17, wherein: The instructions cause the one or more processors to identify one or more differences between the one or more additional container images and the one or more container images with respect to: container alignment; Defocus; strength; intensity changes; motion blur; and / or Imaging sensor noise.

19. The non-transitory computer readable medium of claim 17, wherein: The instructions cause the one or more processors to identify one or more differences between the one or more additional container images and the one or more container images by at least: computing a Fast Fourier Transform (FFT) of the one or more container images or a composite image derived therefrom; and An FFT of the one or more additional container images or a composite image derived therefrom is calculated.

20. The non-transitory computer readable medium of claim 17, wherein: The instructions cause the one or more processors to identify one or more differences between the one or more additional container images and the one or more container images by at least: generating a histogram of pixel intensity levels of the one or more container images or composite images derived therefrom; and A histogram of pixel intensity levels of the one or more additional container images or composite images derived therefrom is generated.

21. The non-transitory computer readable medium of claim 17, wherein: The instructions cause the one or more processors to identify one or more differences between the one or more additional container images and the one or more container images by at least: computing the Laplacian of the one or more container images or composite images derived therefrom; and A Laplacian of the one or more additional container images or a composite image derived therefrom is calculated.

22. The non-transitory computer readable medium of claim 17, wherein: The instructions cause the one or more processors to (i) identify the one or more differences and (ii) generate the visual indication in real time upon receiving the one or more additional container images.

23. The non-transitory computer readable medium of claim 17, wherein: the instructions causing the one or more processors to generate one or more suggestions for modifying the simulated AVI station based on the one or more differences; and The visual indication indicates the one or more suggestions.

24. The non-transitory computer readable medium of claim 23, wherein: The one or more suggestions include: a recommendation for modifying one or more configurable settings of at least one imaging device of the simulated imaging system; a recommendation for modifying one or more configurable settings of at least one lighting device of a simulated illumination system of the simulated AVI station; a suggestion for modifying the position of at least one imaging device of the simulated imaging system; and / or A suggestion for modifying a position of at least one lighting device of the simulated illumination system.

25. A system comprising: Automatic Visual Inspection (AVI) station, the AVI station includes Imaging system, Irradiation system, and sample positioning hardware configured to hold and / or support a container containing a sample; An AVI station that simulates one or more AVI functions of the AVI station, the AVI station comprising Simulation irradiation system, Analog imaging systems, and simulated sample positioning hardware configured to hold and / or support a container containing a sample; as well as a computing system configured to receive one or more images of the vessel captured by the imaging system of the AVI station; receiving one or more additional images of the container captured by the simulated imaging system of the simulated AVI station; identifying one or more differences between the one or more additional container images and the one or more container images; as well as A visual indication is generated of: (i) the one or more differences, and / or (ii) one or more suggestions for modifying the simulated AVI station.

26. The system of claim 25, wherein: The computing system is configured to identify one or more differences between the one or more additional container images and the one or more container images with respect to: container alignment; Defocus; strength; intensity changes; motion blur; and / or Imaging sensor noise.

27. The system of claim 25, wherein: The computing system is configured to identify one or more differences between the one or more additional container images and the one or more container images by at least: computing a Fast Fourier Transform (FFT) of the one or more container images or a composite image derived therefrom; and An FFT of the one or more additional container images or a composite image derived therefrom is calculated.

28. The system of claim 25, wherein: The computing system is configured to identify one or more differences between the one or more additional container images and the one or more container images by at least: generating a histogram of pixel intensity levels of the one or more container images or composite images derived therefrom; and A histogram of pixel intensity levels of the one or more additional container images or composite images derived therefrom is generated.

29. The system of claim 25, wherein: The computing system is configured to identify one or more differences between the one or more additional container images and the one or more container images by at least: high-pass filtering the one or more container images or composite images derived therefrom; as well as The one or more additional container images or a composite image derived therefrom are high pass filtered.

30. The system of claim 29, wherein: The computing system is configured to identify one or more differences between the one or more additional container images and the one or more container images by at least: computing the Laplacian of the one or more container images or composite images derived therefrom; and A Laplacian of the one or more additional container images or a composite image derived therefrom is calculated.

31. The system of claim 25, wherein: The computing system is configured to (i) identify the one or more differences and (ii) generate the visual indication in real time upon receiving the one or more additional container images.

32. The system of claim 25, wherein: The computing system is configured to generate one or more suggestions for modifying the simulated AVI station based on the one or more differences; and The visual indication indicates the one or more suggestions.

33. The system of claim 25, wherein: The one or more suggestions include: a recommendation for modifying one or more configurable settings of at least one imaging device of the simulated imaging system; a recommendation for modifying one or more configurable settings of at least one lighting device of a simulated illumination system of the simulated AVI station; a suggestion for modifying the position of at least one imaging device of the simulated imaging system; and / or A suggestion for modifying a position of at least one lighting device of the simulated illumination system.

Images