Offline troubleshooting and development of automatic visual checkpoint

Abstract

In a method for replicating performance of an Automatic Vision Inspection (AVI) station, a simulated AVI station is constructed that performs one or more AVI functions of the AVI station. One or more container images are captured by an imaging system of the AVI station while the container is illuminated by an illumination system of the AVI station, and one or more additional container images are captured by a simulated imaging system of the simulated AVI station. The method further includes identifying, by one or more processors, one or more differences between the one or more additional container images and the one or more container images; generating, by the one or more processors, a visual indication of the difference (s) and / or one or more suggestions for modifying the simulated AVI station; and modifying the simulated AVI station based on the visual indication.

Description

Technical Field

[0001] This application generally relates to an automated visual inspection (AVI) system for pharmaceutical products or other products, and more specifically to techniques for performing offline troubleshooting and / or development of AVI stations. Background Technology

[0002] In certain situations (such as quality control procedures for manufactured pharmaceutical products), it is necessary to inspect samples (e.g., containers such as syringes or vials, and / or their contents, such as liquid or lyophilized pharmaceutical products) for defects. Under applicable quality standards, the acceptability of a particular sample may depend on indicators such as the type and / or size of vessel defects (e.g., chips or cracks), or the type, quantity, and / or size of undesirable particles (e.g., fibers) within the pharmaceutical product. If a sample has unacceptable indicators, it may be rejected and / or discarded.

[0003] To handle the sheer volume typically associated with commercial pharmaceutical production, defect inspection tasks have become increasingly automated. Furthermore, the specialized equipment used to support automated defect inspection has become extremely large, complex, and expensive, requiring significant investment of human and other resources to identify and debug each new product line. As just one example, consider the filling completion inspection phase for syringes filled with medication. The 296S commercial production line equipment includes 15 individual vision inspection stations and a total of 23 cameras (i.e., one or two cameras per station). Overall, this equipment is designed to detect a wide range of defects, including vessel integrity defects such as large cracks or vessel sealing defects, aesthetic vessel defects such as scratches or stains on the vessel surface, and defects related to the pharmaceutical product itself such as liquid color or the presence of foreign particles.

[0004] Because purchasing add-ons for AVI production line equipment can be costly, troubleshooting and new product characterization activities typically must be carried out on-site. Therefore, troubleshooting and new product characterization often require significant downtime, resulting in suboptimal long-term production rates. Summary of the Invention

[0005] The embodiments described herein relate to systems and methods in which a “simulated” AVI station is constructed or upgraded in an effort to replicate the performance of an existing AVI station, thereby allowing offline troubleshooting or new product characterization and / or qualification work without interfering with or to a minimal extent with production line operation. In some embodiments, a simulated AVI station is a dedicated offline (e.g., lab-based) station that simulates one or more AVI functions of a station in an existing commercial production line facility (e.g., one of a plurality of stations in a production line facility). In such embodiments, a simulated AVI station can be used to troubleshoot problems at a specific corresponding station in a commercial production line facility, or otherwise improve the performance of that corresponding station, without requiring a prolonged shutdown of the production line facility. For example, hardware components (e.g., lighting, star wheels, etc.), hardware layout (e.g., distance or angle between the sample and the camera or lighting, lighting configuration, etc.), and / or software (e.g., code implementing inspection algorithms) can be modified offline. Once appropriate modifications are identified, the commercial production line facility can be shut down for a relatively short period to implement these changes to the original AVI station, possibly followed by a certain amount of field qualification work. Because the AVI simulation station is offline, it provides the opportunity to conduct root cause investigations, formulation development, and / or other support activities in the laboratory rather than on commercial production line equipment.

[0006] In other embodiments, the simulated AVI station is instead a station on a commercial production line, and the goal is to simulate the characteristics / performance of a lab-based AVI station. In such an embodiment, the lab-based AVI station can be used to characterize and validate inspections of new pharmaceutical products, which would otherwise / traditionally require significant downtime of the production line equipment and prevent its simultaneous use with other pharmaceutical products. Once the appropriate hardware components / configuration and suitable software are identified, the production line equipment can be shut down for a relatively short period to implement these changes (again, possibly followed by some amount of field validation work). Similar to the previous embodiment, this embodiment provides the opportunity to conduct formulation development, root cause investigations, and / or other support activities in a laboratory setting rather than on commercial production line equipment.

[0007] In any of these embodiments, constructing a suitable and similar simulated AVI station presents significant challenges. In particular, it is crucial that the imagers (e.g., multiple cameras, imaging optics), illumination (e.g., multiple lighting devices, ambient / surround lighting, or reflections), relative geometry (i.e., spatial arrangement), image processing software, computer hardware, and / or mechanical movement of products (all of which can affect inspection performance) closely match the AVI station being reproduced. This is particularly challenging because many of these components / characteristics tend to be unique to any given AVI station. Therefore, in the embodiments disclosed herein, robust and reliable processes are used to replicate the AVI station as closely as possible (or as closely as desired).

[0008] Initially, any of a variety of suitable technologies can be used to identify the components / constructions of the AVI station. For example, detailed manual photography, 3D scanning, and measurement can be performed. Alternatively or additionally, three-dimensional computer-aided design (CAD) files (e.g., exploded technical drawings in vector graphics PDF or other formats) can be used for this purpose. Using this information, the hardware for simulating the AVI station can be obtained and / or assembled, and said hardware can be placed in the same relative arrangement / geometry as the original AVI station (i.e., the station being simulated). 3D scanners or other devices / technologies can also be used to recreate the AVI station.

[0009] The various techniques disclosed herein can be used to improve and validate a constructed or partially constructed analog AVI station by comparing sample (e.g., container) images captured by the analog AVI station with sample images captured by the AVI station being reproduced. Feedback obtained from this process allows users (e.g., engineers) not only to determine whether the analog AVI station performs in a manner sufficiently similar to the original AVI station, but also to determine which aspects of the analog AVI station should be modified to better replicate the performance of the original AVI station.

[0010] In some embodiments, for this purpose, the image comparison software tool performs comparisons and generates corresponding outputs. For example, the image comparison tool can calculate and report significant image metrics (e.g., metrics indicating light intensity, camera noise, camera / sample alignment, defocus, motion blur, etc.) to the user in real time, thereby providing a reliable process for fine-tuning and evaluating the feasibility of a simulated AVI station in a relatively rapid manner. In some embodiments, the image comparison tool generates specific recommendations based on metrics (e.g., “reduce the distance between the camera and the container”), which are displayed to the user. Advantageously, the image comparison tool can achieve accurate reproduction of AVI station performance even when the original AVI station and the simulated AVI station are located at remote locations. That is, AVI station performance can be adequately reproduced even in cases where it is difficult or impossible to accurately reproduce hardware geometry, computer hardware, and / or other aspects of the AVI station. The image comparison tool typically provides a scientific, repeatable process that reduces the risks associated with human error and subjectivity, and is therefore more likely to meet regulatory requirements regarding true equivalence between the AVI station and the corresponding simulated station. Attached Figure Description

[0011] Those skilled in the art will understand that the accompanying drawings described herein are included for illustrative purposes and not for limiting the scope of this disclosure. The drawings are not necessarily drawn to scale, but rather focus on illustrating the principles of this disclosure. It should be understood that in some cases, aspects of the described embodiments may be shown enlarged or amplified to aid in understanding the described embodiments. Throughout the drawings, similar reference numerals generally refer to components that are functionally similar and / or structurally similar.

[0012] Figure 1A and Figure 1B An example process for troubleshooting AVI stations on commercial production line equipment by building and using simulated AVI stations is described.

[0013] Figure 2 An example process is described for developing AVI recipes and / or hardware settings in an AVI station for commercial production line equipment by simulating laboratory-based settings.

[0014] Figure 3 It is feasible. Figure 1A and Figure 1B A simplified block diagram of an example system for the process.

[0015] Figures 4A to 4D An example depicting an AVI station that can simulate production line equipment or a development platform that can be used as an AVI station for production line equipment is shown, based on a laboratory setting and associated container images.

[0016] Figure 5Another example of a laboratory-based AVI station that can simulate production line equipment or be used as an AVI station for production line equipment is depicted.

[0017] Figure 6 An example image comparison tool is described that can be used to facilitate the construction or upgrading of simulated AVI stations.

[0018] Figure 7 It describes what can be made by Figure 6 The example algorithm implemented by the image comparison tool.

[0019] Figure 8 This is a flowchart of an example method for replicating the performance of an AVI station based on laboratory or production line equipment. Detailed Implementation

[0020] The different concepts described above and discussed in more detail below can be implemented in any of a variety of ways, and the described concepts are not limited to any particular implementation. Examples of implementations are provided for illustrative purposes.

[0021] Figure 1A and Figure 1B An example process 100 is described for troubleshooting automated visual inspection (AVI) stations on commercial production line equipment by constructing and using simulated AVI stations. First, refer to... Figure 1A In phase 102, a commercial production line equipment comprising one or more AVI stations operates in normal / production operation mode. For example, this commercial production line equipment may be used for quality control in the "fill complete" phase for the production of pharmaceutical products (e.g., syringes containing liquid pharmaceutical products or glass vials containing lyophilized pharmaceutical products). The AVI stations may include one or more stations dedicated to container inspection (e.g., syringes, vials, etc.) and / or one or more stations dedicated to sample inspection (e.g., detecting and / or characterizing particles in the pharmaceutical product within the container). For example, this commercial production line equipment may be, as referred to below... Figure 3 Commercial production line equipment 302 is discussed in more detail. Figure 1A and Figure 1B The top horizontal line / arrow extending from stage 102 to stage 142 (discussed below) indicates uninterrupted product line inspection using commercial production line equipment. Figure 1A and Figure 1B The horizontal axis usually represents time, but it is not necessarily drawn to scale. Figure 1A and Figure 1B It may not (but may) indicate the order of operations (e.g., stage 110 may occur before or after the first iteration of stage 122, etc.).

[0022] In phase 104, problematic AVI stations within commercial production line equipment are identified. For example, an individual monitoring the production process may observe that a particular AVI station on the production line equipment is identifying a large number of false positives (e.g., samples are marked as defective by the production line equipment but are determined to be acceptable after closer manual or automated inspection), and / or fails to identify defective samples.

[0023] In phase 110, the operator downloads and / or installs the software code for the problematic AVI station into a computing system associated with a lab-based setup (i.e., a simulated AVI station). This code may be transferred directly from the production line equipment or installed in another manner (e.g., downloaded from a portable storage device or the internet). In some embodiments, the installed code includes code responsible for container movement, image capture, and image processing. For example, this code may control mechanisms to shake (e.g., rotate, rock, invert, etc.) the container before and / or during imaging, trigger one or more cameras at appropriate times, and process camera images to detect defects in the container (e.g., cracks, gaps) and / or defects in the contents (e.g., large fibers or other foreign matter).

[0024] In phase 112, the problematic AVI station within the production line equipment captures one or more images of the container. Depending on the embodiment and / or scenario, phase 112 may or may not require any interruption to the normal / production operation of the production line equipment. For example, the captured images may be images that are also used during production.

[0025] In stage 114, the hardware of the problematic AVI station is reverse engineered to initiate a simulated AVI station setup procedure 120. Stage 114 may include reverse engineering the hardware components of the problematic AVI station (e.g., camera, optics, lighting, mechanisms for moving the container, etc.), the assembly of the hardware components of the problematic AVI station (e.g., how the various components and sub-components are assembled), the relative geometry / arrangement of the hardware components in the problematic AVI station (e.g., the orientation and distance of the container relative to (multiple) lighting devices and (multiple) cameras) and / or other characteristics of the problematic AVI station (e.g., container rotation speed, etc.). In some embodiments, the reverse engineering is purely manual and involves precise measurements (e.g., calipers, rulers, etc.), review of available schematics, etc. A 3D scanner may also be used to accurately capture the dimensions of the AVI station. In other embodiments, at least a portion of the reverse engineering may be automated, for example, by processing files or images indicating the dimensions (angles, distances, etc.) of the problematic AVI station.

[0026] In the first iteration of Phase 122, the knowledge gained in Phase 114 is used to construct a simulated AVI station. This construction can be partially or entirely manual. Any suitable fabrication techniques (such as CNC machining of metals and plastics, and / or 3D printing) can be used to construct certain non-electronic hardware components of the simulated AVI station (e.g., star wheels, etc.). The first iteration of Phase 122 may also involve purchasing or otherwise acquiring various off-the-shelf components, such as cameras, LED rings, or other lighting devices. The first iteration of Phase 122 may also involve setting various software parameters to match the parameter settings used in Phase 112. For example, the user can set the container rotation speed to be equal to the rotation speed setting used by the production line equipment when capturing (multiple) images in Phase 112.

[0027] In the first iteration of phase 124, after the initial attempt (in phase 122) to replicate the problematic AVI station, one or more images of the container are captured by one or more imagers (e.g., cameras) simulating the AVI station. This container should be of the same type as the container imaged in phase 112, and in fact, can be the same container.

[0028] In the first iteration of phase 126, the image comparison tool determines whether the container(s) captured in phase 112 match, to some acceptable degree, the container(s) captured in the first iteration of phase 124. To make this determination, the image comparison tool may generate multiple metrics for each image or set of images and compare these metrics to determine a measure of similarity (e.g., a similarity score). For example, the image comparison tool may generate metrics related to size (e.g., how large the container appears within the image), orientation (e.g., the angle of the container walls relative to the vertical axis of the image), light intensity (e.g., as indicated by image pixel intensity), defocus, motion blur, and / or other characteristics. The image comparison tool may also compare corresponding metrics from the images(s) of phase 112 with corresponding metrics from the images(s) of phase 124 (e.g., by calculating differences). See below for reference. Figure 6 and Figure 7 The example metrics are discussed in further detail. Depending on the implementation, the determination in stage 126 can be made by a user observing the output of the image comparison tool or by the tool itself.

[0029] If the image comparison tool (or its user) determines in the first iteration of stage 126 that an image or set of images is not sufficiently similar, the simulated AVI station is modified in the second iteration of stage 122. Based on the output of the image comparison tool in the first iteration of stage 126, the modification in the second iteration of stage 122 is made in a focused manner. For example, if the image comparison tool indicates that the light intensity of (or more) images captured by the simulated AVI station in the first iteration of stage 124 is too low, the user can move the illumination device closer to the container, or change the lens aperture, etc., during the second iteration of stage 122. As another example, if the image comparison tool indicates that the image captured in the first iteration of stage 124 is less focused than the image captured in stage 112, the user can move the container closer to or further away from the camera of the simulated AVI station. In some embodiments, the image comparison tool processes metrics of the compared images to provide suggestions in stage 126, such as "move illumination device B closer to the container," "move illumination device B closer to the container," or "move illumination device B 3mm closer to the container," etc.

[0030] After the developers make (multiple) modifications in the second iteration of phase 122, the simulated AVI station captures a new set of one or more images in the second iteration of phase 124 (e.g., in response to manual triggering by the user), and the image comparison tool compares (multiple) new images with the images captured in phase 112 (or possibly with new images captured by the problematic AVI station) in the second iteration of phase 126. Program 120 (as...) Figure 1A The loop within (shown) can continue for any number of iterations until the image comparison tool (or the user observing its output) determines in the iteration of phase 126 that the simulated AVI station has sufficiently accurately reproduced the performance / characteristics of the problematic AVI station, as indicated by sufficient similarity between the images(s) captured by the simulated AVI station and the images(s) captured by the problematic AVI station from the production line equipment. In some scenarios, sufficient similarity can be achieved even with significant differences in hardware components and / or geometry. In other scenarios, sufficient similarity requires precise replication of hardware components and geometry.

[0031] When sufficient similarity is achieved, in process 130 (e.g.) Figure 1BDuring this period (as shown), the simulated AVI station is prepared for troubleshooting capabilities. In example process 130, in the first iteration of phase 132, the user (e.g., an engineer) considers / theorizes appropriate modifications to the inspection algorithm / recipe, and / or modifications to hardware settings (e.g., different camera, lens, lighting device types, etc., and / or different arrangements and / or settings of such devices / components) in an attempt to correct the problem observed in phase 104. Subsequently, in the first iteration of phase 134, the operator modifies the hardware and / or code based on the modifications identified in the first iteration of phase 132.

[0032] In the first iteration of Phase 136, the simulated AVI station is used to test whether its performance is satisfactory, i.e., whether the problems observed in Phase 104 have been corrected to a sufficient degree. For example, Phase 136 may involve comparing statistical results (e.g., false positive rate, etc.) with standards-based requirements. Each iteration of Phase 136 may be time-intensive and / or labor-intensive, as it may require a large number of images and / or a wide variety of container / product samples to determine whether the problem has been resolved (e.g., if the observed problem is a low but still acceptable false positive or false negative rate). However, the time investment can be acceptable, as it does not require interruption of commercial production line equipment.

[0033] If performance is not determined to be satisfactory / acceptable in stage 136, process 130 is repeated, in which new modifications are identified / theorized in the second iteration of stage 132. Process 130 can be repeated for any number of iterations without interrupting the operation of the production line equipment until performance is determined to be satisfactory / acceptable in the iteration of stage 136. At this point, troubleshooting process 130 is completed, and if the qualification and commissioning activities (in stage 140) have been successfully performed, the modifications made during process 130 (i.e., as reflected in the final state of the simulated AVI system after the final iteration of stage 134) are applied to the problematic AVI station in stage 142. While stage 142 typically requires stopping production on the commercial production line equipment to make changes from process 130 (and possibly some brief qualification / commissioning operations), the downtime is significantly shorter than the downtime that would otherwise have to be performed on-site at the problematic AVI station itself during troubleshooting process 130. After the changes are applied to the problematic AVI station in stage 142, production (i.e., normal / production operation of the production line equipment) resumes in stage 144.

[0034] Although process 100 has been described with reference to troubleshooting problematic AVI stations, it should be understood that process 100 can be used alternatively to improve AVI stations that are already performing well (e.g., to further optimize the inspection performance of the AVI station or to make the AVI station more cost-effective, etc.). Furthermore, although process 100 has been described with reference to the filling completion stage of a pharmaceutical production line, it should be understood that process 100 can be used alternatively at different stages (e.g., when inspecting products after equipment assembly, or when inspecting product labeling and / or packaging, etc.), and / or alternatively in non-pharmaceutical situations (e.g., in another situation with relatively stringent quality standards).

[0035] Figure 1A and Figure 1B A process 100 for troubleshooting AVI stations on commercial production line equipment by constructing and using simulated AVI stations is described, while Figure 2 Example process 200 is described, in which an AVI station of commercial production line equipment is upgraded to simulate the performance of laboratory-based AVI settings. For example, Figure 2 Commercial production line equipment includes one or more AVI stations, and can be a combination of the above. Figure 1A and Figure 1B Or refer to the following text Figure 3 Any type of production line equipment discussed (i.e., commercial production line equipment 302).

[0036] In phase 202, commercial production line equipment operates in normal / production mode, for example, to perform a filling completion phase check for a specific pharmaceutical product (e.g., a syringe filled with the drug). (As described above...) Figure 1A and Figure 1B The subject of discussion Figure 2 Process 200 can be applied alternatively to different inspection stages (e.g., device assembly, packaging, etc.), and / or process 200 can be used in non-pharmaceutical situations. Figure 2 The top horizontal line / arrow extending from stage 202 to (discussed below) stage 220 indicates uninterrupted product line inspection using commercial production line equipment. Figure 2 The horizontal axis usually represents time, but it is not necessarily drawn to scale. Figure 2 It may not (but may) indicate the order of operations (e.g., phase 202 may begin before or after phase 204).

[0037] In stage 204, a decision is made to adapt the commercial production line equipment for the fill completion inspection of the new drug product. For various reasons, the new product may require customized modifications to one or more AVI stations on the commercial production line equipment. For example, the new drug product may have lower transparency than previous products (e.g., requiring greater light intensity for imaging), or it may be placed in different types of containers with different types and / or areas of potential defects.

[0038] Next, in development procedure 210, a laboratory-based AVI station is developed to be customized for the new drug product. Within development procedure 210, in stage 212, an imaging system (one or more cameras and any associated optical components) based on the laboratory setting captures images of the irradiated sample (e.g., a fluid or lyophilized product) in a container (e.g., a syringe or vial).

[0039] In phase 214, using the captured images, the user develops an inspection formula / algorithm based on laboratory settings and tunes various parameters based on those settings to strive for desired performance (e.g., false positives and / or false negatives of (multiple) specific types of defects being less than a threshold amount). Tuned "parameters" can include any settings, type, location, and / or other characteristics of software, imaging hardware, lighting hardware, and / or computer hardware. For example, the user can adjust light intensity settings, camera settings, camera lens type or other optical components, the geometry of the imaging and illumination systems, etc. It should be understood that the term "user" as used throughout this disclosure can refer to a single individual or a team of two or more individuals. In some scenarios, the user may develop entirely new software algorithms in phase 214.

[0040] In phase 216, the user performs characterization and evaluation work to determine whether the lab-based setup / station, as developed / tuned in phase 214, has performed satisfactorily (e.g., according to applicable specifications). If not, further development / tuning can be performed using the lab-based setup / station in another iteration of phase 214, which may also require capturing additional images in another iteration of phase 212. Phases 212, 214, and 216 of development procedure 210 can be repeated for any number of iterations until the results of the characterization / evaluation work in phase 216 are considered satisfactory.

[0041] When the results are deemed satisfactory and the new product is ready for commercial-scale production, the commercial production line equipment is shut down, and in Phase 220, the hardware and / or software of the AVI station on the production line equipment are updated to simulate performance based on laboratory settings. Although not in Figure 2 The document explicitly states that, however, the simulation process within phase 220 may involve... Figure 1AThe iterative setup procedure 120 (i.e., stages 122, 124, 126) is similar, but since an AVI site already exists and is being updated, the initial iteration of stage 122 may (not) require any "build". In fact, in some scenarios, the first iteration of stage 122 can be skipped entirely (i.e., if the developers believe that the existing AVI site is close enough to the lab-based setup to proceed directly to the image comparison tool stage) and only executed for subsequent iterations.

[0042] It may also require developers to perform some level of field qualification work in phase 220 (after a successful update based on the image comparison tool), which would increase downtime on the production line equipment. However, this would likely require far less time than the qualification work during development procedure 210, and in any case, the downtime on the production line equipment would be significantly reduced due to the fact that development work (in procedure 210) occurs offline. In phase 222, following successful qualification, the commercial production line equipment resumes production, but now it is producing the new product.

[0043] In some scenarios, both Process 100 and Process 200 are implemented sequentially. For example, if it is determined that the AVI station on the production line equipment is generating an unacceptable number of false positives (i.e., marking such defects when no major defects exist), Process 100 can be used to build and tune the simulated AVI station. Subsequently, by using the simulated AVI station, it can be determined that different hardware components are needed (e.g., one LED ring light instead of multiple directional lights). After the new design for the simulated AVI station has been identified, Process 200 can be used when upgrading the AVI station in the production line equipment to ensure that the upgraded station's performance is accurately / sufficiently matched to the simulated station.

[0044] Figure 3 This is a simplified block diagram of an example system 300 that can implement the techniques described herein. Specifically, Figure 3 An embodiment of troubleshooting AVI stations for commercial production line equipment using a simulated AVI station is described. Therefore, for example, system 300 can be implemented and / or used to implement... Figure 1A and Figure 1B The process 100.

[0045] like Figure 3 As shown, system 300 includes commercial production line equipment 302 and simulated AVI stations 304. Production line equipment 302 can be any production-grade equipment with N (N≥1) AVI stations 310-1 to 310-N (also collectively referred to as AVI stations 310). As just one example, production line equipment 302 can be... 296S production line equipment. In Figure 3In the example, the i-th AVI station 310-i of production line equipment 302 requires troubleshooting (or alternatively, optimization), where i equals 1, N, or any number between 1 and N. Each of the AVI stations 310 can be responsible for capturing images of different aspects to be used to inspect containers and / or samples within containers. For example, the first AVI station 310-1 can capture a top-view image of a syringe or vial to check for cracks or nicks, and the second AVI station 310-2 can capture a side-view image to check for foreign particles in the contents of the syringe or vial (e.g., fluid or lyophilized pharmaceutical products).

[0046] Figure 3 The typical components of the i-th AVI station 310-i are shown in simplified block diagram form. In particular, AVI station 310-i includes an imaging system 312, an illumination system 314, and sample positioning hardware 316. It should be understood that other AVI stations 310 (if any) may be similar, but may have different component types and configurations, which are appropriate for the purpose of each given station 310.

[0047] Imaging system 312 includes one or more imaging devices and may include associated optical components (e.g., additional lenses, mirrors, filters, etc.) to capture an image of each sample (e.g., containers and pharmaceutical products). For example, these imaging devices may be cameras with charge-coupled device (CCD) sensors. As used herein, the terms "camera" or "imaging device" can refer to any suitable type of imaging device (e.g., a camera that captures a portion of the visible spectrum, or an infrared camera, etc.). For example, illumination system 314 includes one or more illumination devices, such as an array of light-emitting diodes (LEDs) (e.g., a panel device or a ring device), for illuminating each sample for imaging.

[0048] The sample positioning hardware 316 may include any hardware that holds or otherwise supports the sample, and may include hardware for transporting and / or otherwise moving the sample for the AVI station 310-i. For example, the sample positioning hardware 316 may include a star wheel, a disc conveyor, a robotic arm, etc. In some embodiments, depending on the function of the AVI station 310-i, the sample positioning hardware 316 may also include hardware for shaking each sample. For example, if the AVI station 310-i is checking for foreign particles in a liquid, the sample positioning hardware 316 may include components for rotating, inverting, and / or shaking each sample.

[0049] Production line equipment 302 also includes a processing unit 320 and a memory unit 322. The processing unit 320 may include one or more processors, each of which may be a programmable microprocessor that executes software instructions stored in the memory unit 322 to perform some or all of the software-controlled functions of the production line equipment 302 as described herein. Alternatively or additionally, some processors in the processing unit 320 may be other types of processors (e.g., application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc.), and a function of the processing unit 320 as described herein may alternatively be implemented in hardware. The memory unit 322 may include one or more volatile and / or non-volatile memories. The memory unit 322 may include any suitable one or more memory types, such as read-only memory (ROM), random access memory (RAM), flash memory, solid-state drives (SSDs), hard disk drives (HDDs), etc. The memory unit 322 may collectively store one or more software applications, data received / used by these applications, and data output / generated by these applications.

[0050] Processing unit 320 and memory unit 322 are jointly configured to control / automate the operation of AVI station 310 and process images captured / generated by AVI station 310 to detect defects of a corresponding type in containers and / or container contents (e.g., pharmaceutical products). In an alternative embodiment, the functions of processing unit 320 and / or memory unit 322 are distributed among N different processing units and / or memory units, each specific to a different AVI station from AVI station 310-1 to 310-N. In yet another embodiment, one function of processing unit 320 and memory unit 322 (e.g., for sample transport, shaking, and / or imaging) is distributed among AVI stations 310, while other functions of processing unit 320 and memory unit 322 (e.g., for processing sample images to detect defects) are performed by a centralized processing unit. In some embodiments, at least a portion of the processing unit 320 and / or the memory unit 322 is included in a computing system (e.g., a specially programmed general-purpose computer) located outside (and possibly far from) the production line equipment 302.

[0051] Memory unit 322 stores the sample (container / product) images captured by AVI station 310, and also stores AVI code 326 that, when executed by processing unit 320, performs the following two operations: (1) causing AVI station 310 to capture images; and (2) processing the captured images to detect defects (e.g., as discussed above). For AVI station 310-i, for example, AVI code 326 includes... Figure 3The corresponding portion of code 328 is represented in the text. As an example of an embodiment, code 328 may trigger imaging system 312 to capture images while the sample is being irradiated by irradiation system 314, and may control sample positioning hardware 316 to place the sample in the correct position at the appropriate time, and may shake the sample according to a shaking curve at the appropriate time. After capturing images and storing these images as images 324, code 328 processes images 324 to detect specific types of defects associated with station 310-i. As described above, in some embodiments, the portion of code 328 that processes images may be performed by a different processor, component, and / or device than the portion of code 328 that controls imaging, shaking, etc.

[0052] The simulated AVI station 304 may be a laboratory-based setting constructed to attempt to replicate (to a sufficient extent) the performance of a specific AVI station 310-i (e.g., in response to learning that the AVI station 310-i has unacceptable levels of false positives or false negatives). The simulated AVI station 304 includes a simulated imaging system 332, a simulated illumination system 334, and simulated sample positioning hardware 336. The simulated imaging system 332 includes one or more imaging devices (and may include associated optics) for capturing images of each sample (e.g., containers and pharmaceutical products), the simulated illumination system 334 includes one or more illumination devices for illuminating each sample for imaging, and the sample positioning hardware 316 includes hardware for holding or otherwise supporting the sample, and may include hardware for transporting and / or otherwise moving the sample.

[0053] Ideally, the simulated imaging system 332, simulated illumination system 334, and simulated sample positioning hardware 336 will perfectly replicate the imaging system 312, illumination system 314, and sample positioning hardware 316 of the AVI station 310-i, respectively. More importantly, the simulated AVI station 304 as a whole will ideally replicate the performance of the AVI station 310-i. However, in the real world, precise performance matching is very difficult to achieve. As mentioned above... Figure 1A As described above, various manual and / or automated reverse engineering techniques can be used to construct a simulated AVI station 304 that initially has performance “close” to that of AVI station 310-i (e.g., in stage 122).

[0054] After initially building the simulated AVI site 104, as referenced above... Figure 1AAs discussed in stage 126, software can be used to facilitate fine-tuning of the analog AVI station 304 to improve performance matching. For this purpose, the analog AVI station 304 is coupled to a computing system 340 (e.g., a specially programmed general-purpose computer), which includes a processing unit 342 and a memory unit 344. The computing system 340 may be separate from or integrated with the analog AVI station 304, and may be located near or away from the analog AVI station 304. In some embodiments, for example, the computing system (or a portion thereof) receives images from the analog AVI station 304 via an internet link.

[0055] Processing unit 342 may include one or more processors, each of which may be a programmable microprocessor that executes software instructions stored in memory unit 344 to perform some or all of the software-controlled functions of computing system 340 as described herein. Alternatively or additionally, some processors in processing unit 342 may be other types of processors (e.g., ASIC, FPGA, etc.), and a function of processing unit 342 as described herein may alternatively be implemented in hardware. Memory unit 344 may include one or more volatile and / or non-volatile memories. Memory unit 344 may include any suitable one or more memory types, such as ROM, RAM, flash memory, SSD, HDD, etc. Memory unit 344 may collectively store one or more software applications, data received / used by these applications, and data output / generated by these applications.

[0056] Memory unit 344 stores images 346 captured by analog imaging system 332 and images 348 captured by imaging system 312 of AVI station 310-i. Memory unit 344 also stores an image comparison tool (ICT) 350 and AVI code 352. Typically, image comparison tool 350 facilitates the process of tuning analog AVI station 304 so that its performance matches that of AVI station 310-i (e.g., as described in combination stage 126 above and below). Figure 6 and Figure 7 (As discussed), and AVI code 352 is used to control the constructed and tuned analog AVI station 304 during troubleshooting or optimization processes. For example, AVI code 352 may be a perfect (or very close) copy of code 328 and may be downloaded or uploaded from production line equipment 302, from a portable storage device, from the Internet, or from another suitable source. In other embodiments, AVI code 352 includes only a portion of code 328 (e.g., excluding the portion used to control the transport of samples to and from the appropriate imaging position).

[0057] The computing system 340 is coupled to the output unit 360, which can be any type of visual and / or audio output device (e.g., a computer monitor, touchscreen, or other display and / or speaker of the computing system 340, or a separate computing device having a display and / or speaker and coupled to the computing system 340, etc.). The image comparison tool 350 and AVI code 352 enable the output unit 360 to provide various visual and / or audio outputs to the developers or users of the analog AVI station 304. For example, the image comparison tool 350 enables the output unit 360 to display various indicators representing the differences between image 346 and image 348 (as discussed further below), and the AVI code 352 enables the output unit 360 to display information such as indicators of whether a particular sample has defects.

[0058] Although Figure 3 An embodiment is described using a simulated AVI station (station 304) to troubleshoot the AVI station (station 310-i of production line equipment 302) of a commercial production line device. However, it should be understood that similar components can be used to update / modify the AVI station of the commercial production line device to replicate the performance of laboratory-based settings / stations (e.g., as in...). Figure 2 In an embodiment of process 200, the i-th AVI station 310-i is a “simulation” station (i.e., based on laboratory settings) that simulates AVI station 304. Furthermore, in such an embodiment, the image comparison tool 350 may alternatively reside in the memory 322 of the production line equipment 302. Alternatively, the image comparison tool 350 may be stored in the computing system 340.

[0059] In some embodiments, system 300 provides access to remote sites (e.g., global manufacturing sites) so that remote users can directly access lab-based settings, regardless of the specific lab and manufacturing location. This approach enables real-time collaboration between sites / users across networks and allows troubleshooting and / or development support to be located at centralized global facilities, while still leveraging the expertise of individuals (engineers, etc.) located in various locations. Therefore, this networked approach can lead to more efficient organizational structures.

[0060] Figure 4 and Figure 5 Examples of laboratory-based settings are described, any of which can be Figure 3 The simulated AVI station 304 (e.g., the simulated AVI station of process 100) or alternatively, a lab-based AVI station used as a development platform (e.g., the lab-based setup used in process 200). It should be understood that these lab-based setups / stations are merely illustrative, and there is an almost unlimited number of alternative types and configurations.

[0061] First refer to Figure 4A For example, a first lab-based setup 400 can be used to inspect the top of a container (e.g., the top of a syringe filled with a liquid pharmaceutical product) for defects (e.g., cracks, gaps, etc.). Setup 400 includes a camera 402 for imaging the containers and an LED ring 404 for illuminating each container while simultaneously imaging it. Sample positioning hardware 406 includes a platform 406A and a star wheel 406B mounted on the platform 406A. The camera 402 and LED ring 404 may also be mounted (directly or indirectly) on the platform 406A. The star wheel 406B can hold the container (e.g., a syringe) in fittings along its periphery, and the star wheel 406B rotates relative to the platform 406A to move each container into an imaging position (i.e., centered within the LED ring 404 and directly below the camera 402). However, in some embodiments and / or scenarios, the only important aspect is setting the AVI station 400, which is matched or closely proximate to the production line equipment, to image a single container (e.g., lighting, optics, etc.), rather than being able to sequentially image multiple consecutive containers. Therefore, in some embodiments, even if the AVI station being simulated (or the AVI station under development) requires such hardware or functionality, the star wheel 406B does not need to be designed to hold multiple containers or rotate.

[0062] Figure 4B An example image 450 that can be captured by setting 400 is depicted for capturing a top-view image of a syringe for detecting defects on the syringe flange, with regard to setting 400 (and the corresponding AVI station of the production line equipment). As shown in this example, these defects can include notches and cracks of various locations and sizes on the syringe flange.

[0063] Figure 4C The AVI station representing the equipment on the production line (e.g., is described) is depicted. Figure 3 Example image 460 of the images captured by the AVI station 310-i), while Figure 4D Example image 470 depicts an image captured by a simulated AVI station (e.g., setting 400) after successful troubleshooting. In this example, the production line equipment AVI station may have an unacceptably high false positive rate due to light reflections as shown in image 460, and modifications made during troubleshooting (e.g., changing from another type of lighting fixture to an LED ring, or changing the orientation of the camera or(s) lighting fixtures relative to the container, etc.) significantly reduce reflections, as shown in image 470. Reduction of artifacts related to light reflections can reduce the likelihood of defects being masked, and / or reduce the likelihood of reflections being misinterpreted as defects.

[0064] Now for reference Figure 5For example, a lab-based setup 500 could be used to inspect syringes for defects (e.g., cracks, nicks, stains, etc. in the sidewall of the syringe container, and / or unacceptable types and / or quantities of particles within the pharmaceutical product) from a side view (e.g., the side of a syringe filled with a liquid pharmaceutical product). Setup 500 includes a camera 502 for imaging the container and an LED backlight 504 for illuminating each container from behind while imaging it. Sample positioning hardware includes a disc conveyor 506 that holds multiple syringes and positions individual syringes between the camera 502 and the LED backlight 504. The disc conveyor 506 can rotate to move each syringe into the imaging position. As described above, in some embodiments and / or scenarios, the only important aspect is that setup 500 matches or closely approximates the AVI stand of the production line equipment in terms of imaging a given sample rather than having the same ability to sequentially image multiple consecutive samples. Therefore, in some embodiments, even if the AVI station being simulated (or the AVI station under development) requires such hardware or functionality, the disc conveyor 506 does not need to be designed to hold multiple syringes or rotate.

[0065] Figure 6 The diagram depicts a laboratory-based setup that can be used to facilitate the simulation of AVI stations (e.g., in process 100 of Figure 1). Figure 2 The AVI station or process 200 production line equipment in the middle Figure 3 Example image comparison tool 600 for building and / or upgrading simulated AVI site 304 (e.g., Figure 3 Image comparison tool 350). For example... Figure 6 As shown, the image comparison tool 600 receives a raw image 602 and a simulated image 604, wherein the raw image 602 may be an image captured by an AVI station of a production line device and the simulated image 604 may be an image captured by a laboratory-based setting, or vice versa, depending on whether process 100 or process 200 is being implemented. In some embodiments, it is assumed that images 602 and 604 use a lossless image compression format.

[0066] Image comparison tool 600 includes an index generation unit 612 and a feedback unit 614. The index generation unit 612 processes images 602 and 604, and generates / calculates indices indicating the characteristics of images 602 and 604, and calculates one or more additional indices indicating differences between images 602 and 604 based on these indices. In some embodiments, the index generation unit 612 calculates the indices(s) on a per-image basis, i.e., by comparing a single original image in original image 602 with a single simulated image in simulated image 604. In other embodiments, the index generation unit 612 calculates each of these indices based on a set of multiple original images 602 and multiple simulated images 604. For example, after applying alignment techniques, a set of x images (x>1) of original image 602 may be averaged or superimposed on each other, and a set of x images of simulated image 604 may be averaged or superimposed on each other. For example, such a method can reduce the influence of outlier images. The averaging / superimposition / alignment can be performed by, for example, the same computing device or processor implementing image comparison tool 600, or by another device or processor. In alternative embodiments (e.g., if a line scan camera is used), x images of the original image 602 are stitched together, and x images of the simulated image 604 are stitched together, before unit 612 calculates any metrics (e.g., in an embodiment where each container is rotated to obtain a 360-degree view of the container). Further preprocessing of images 602 and 604 is also possible (e.g., generating an image across all x images where each pixel has the maximum intensity at that pixel location, etc.). For ease of explanation, Figure 6 The rest of the description and Figure 7 The description only mentions an original image 602 and a simulated image 604. However, it should be understood that the above changes or other suitable changes are possible. See below for reference. Figure 7 This section discusses some specific examples of the indicators that the indicator generation unit 612 can calculate.

[0067] Based on the multiple difference metrics generated by the metric generation unit 612, the feedback unit 614 presents one or more outputs to the user (e.g., engineers, developers, technicians, etc.) in visual and / or audio formats. For example, these outputs may be generated by… Figure 3 The output unit 360 generates (e.g., displays and / or emits) the generated metrics. In some embodiments, the feedback unit 614 presents only (e.g., on a graphical user interface (GUI)) the generated metrics to the user. For example, the feedback unit 614 may display intensity level metrics, image defocus metrics, and / or other metrics to the user.

[0068] In other embodiments, the feedback unit 614 may alternatively or additionally generate one or more user suggestions based on one or more of these metrics. For example, the feedback unit 614 may be used to generate suggestions such as moving the camera, moving the lighting device, changing the light intensity, changing the camera settings, etc. See below for further details. Figure 7 Discuss other suggested examples.

[0069] In some embodiments, the index generation unit 612 and the feedback unit 614 operate substantially in real time when the simulated image 604 is captured by the imaging system of the simulated AVI station (e.g., by the simulated imaging system 332) and / or when the simulated image 604 is received by an apparatus or system implementing the image comparison tool 600 (e.g., by the computing system 340). Thus, for example, a user may be able to capture a simulated station image by selecting an interactive control on the GUI presented on the output unit 360, and then view the corresponding indexes and / or suggestions generated by the index generation unit 312 and / or the feedback unit 314 almost immediately. In this way, the user can quickly experience iterative modifications to the simulated AVI station (e.g., slight adjustments to component positions, settings, etc.) and observe the impact of these modifications on the performance of the simulated AVI station (i.e., how the modifications may bring the simulated AVI station closer to or further away from the performance of the original AVI station).

[0070] For example, Figure 7 It describes what can be made by Figure 6 Example algorithm 700 implemented by image comparison tool 600 (and more specifically, by index generation unit 612). Figure 7 As shown, algorithm 700 accepts a single original image 702 (e.g., an image in image 602) and a single simulated image 704 (e.g., an image in image 704) as input. As mentioned above, in some embodiments, these “single” images may be a composite (e.g., an average) of multiple images. In one embodiment, algorithm 700 is implemented in Python code using the OpenCV library.

[0071] In stage 712, algorithm 700 determines one or more total image parameters (P1 to Pj, where j≥1) for the original image 702 and the simulated image 704, and compares these parameters to check for consistency. For example, these total image parameters may represent relatively basic image parameters such as image size, image resolution, color depth, and / or image file format. For example, image comparison tool 600 can determine the total image parameters using various hardware components (e.g., cameras) input to the original and simulated AVI stations, or configuration files generated by these hardware components. Algorithm 700 can be implemented using a GUI (e.g., by...). Figure 3The output unit (360° display) indicates any differences in the total image parameters. Therefore, for example, a user may be able to easily detect when the camera (or subsequent processing) in the original AVI station changes the captured image in some way (e.g., cropping, format conversion, resizing, etc.) and can manually reconfigure the simulated AVI station to reproduce these image processing operations.

[0072] In stages 714A and 714B, algorithm 700 calculates one or more metrics (C1 to Ck, where k ≥ 1) for the original image 702 and the simulated image 704, respectively. Stages 714A and 714B may assume that the total image parameters of images 702 and 704 are the same. These metrics may represent any image characteristics that are related to or may be related to the accuracy of the inspection. Algorithm 700 also includes a metric indicating the difference between corresponding metrics of the two images 702 and 704 (in... Figure 7 The difference is denoted as ΔC1 to ΔCk. The algorithm can calculate some or all of the differences, for example, by simple subtraction (e.g., where ΔC1 is equal to the absolute value of the difference between C1 of image 702 and C1 of image 704) or by using other techniques (e.g., element-wise subtraction, dot product, etc.).

[0073] As an example, these metrics may include one or more light intensity metrics. Depending on the embodiment, light intensity may be measured in one or more ways. For example, for each of images 702, 704, algorithm 700 may average the intensity values ​​of all pixels to generate an average value, and compare the average values ​​of images 702, 704 (e.g., by subtraction). As another example, algorithm 700 may generate a pixel intensity histogram for each of images 702, 704, and then use known techniques to perform a mathematical comparison of these histograms. Some techniques that can be used and provide a single-value output reflecting the differences between histograms include the Patacharian distance method, correlation method, Kal's method, and intersection method. The best technique to use may depend on the nature of the images under consideration. Histogram-based intensity analysis allows algorithm 700 to take into account the dynamic intensity range within each image (i.e., the breadth between the minimum and maximum intensity values).

[0074] For complex images, intensity metrics (such as those discussed above) may be insufficient because uneven illumination can reflect disproportionately from a subset of the container's facets and its immediate environment. Furthermore, coarseness effects resulting from uneven ambient illumination (e.g., from a window or fluorescent ceiling light) can lead to large, gradual intensity variations across the image. In some embodiments, algorithm 700 uses a low-pass frequency filter to capture such variations. Alternatively, algorithm 700 may compute a Fast Fourier Transform (FFT) for each of images 702, 704 and process the corresponding FFT outputs to determine whether the frequency content is similarly distributed across images 702, 704.

[0075] As another example, these metrics may include one or more metrics indicating image defocus. For example, algorithm 700 may compute the Laplacian operator for each of images 702 and 704 to generate single-valued parameters that can be easily compared. For example, if algorithm 700 is implemented as Python code using the OpenCV library, the defocus can be computed for each of images 702 and 704 as follows:

[0076] Defocus=cv2.Laplacian(image,cv2.CV_64F).var()

[0077] The Laplacian operator index can also indicate the motion of the imaged container / sample relative to (multiple) imaging cameras. In commercial AVI systems, it is common to move the sample rapidly relative to the inspection station. To achieve sharp imaging, commercial systems typically employ, for example, short camera exposure times, strobe light from the illumination lamp, motion of the imaging components used to track the part, or a combination of one or more of these features. If one or more of these features of the simulated AVI station do not match those of the original AVI station, some degree of motion blur or image trailing may occur. The result is similar to defocus, but with a directional component. Therefore, the index calculated using the Laplacian operator technique can also indicate motion. In addition to the Laplacian operator, first-order filters such as Sobel or Prewitt can be used to obtain additional information about image sharpness.

[0078] As another example, these metrics can include one or more indicators of camera noise. For modern digital industrial cameras, noise can only occur within the sensor itself. Once the signal is digitized, it is generally no longer susceptible to damage. Since camera brands and models can be easily matched in some cases when setting up analog AVI stations, noise levels can be similar. As an example, for a particular camera brand and model, the cumulative noise level on a given pixel in an 8-bit grayscale image can be in the range of 0 to 10 within a dynamic range of 0 to 255. This makes noise at room temperature negligible for most AVI applications. However, it is conceivable that in some cases, camera noise levels can affect inspection performance and / or make it impossible to find and use a camera model due to obsolescence. For digital images, camera noise is typically high-frequency in nature. Therefore, algorithm 700 can generate indicators of camera noise by calculating an FFT for each of images 702, 704 and processing the FFT output to generate an indicator of the level or relative level of high-frequency components.

[0079] As another example, these metrics may include one or more metrics indicating the alignment and scaling of the imaging container (and possibly other objects being imaged, such as part of the sample positioning hardware). For example, algorithm 700 may determine the relative rotation of the imaging container (e.g., the angle of the vertical wall of the imaging container relative to the vertical or horizontal axis of the image itself (i.e., relative to the axis established by the camera frame)) and the lateral and / or scale depth offset / shift (e.g., determined based on length and / or width as measured in pixels).

[0080] In stage 716, algorithm 700 generates a weighted comparison score for image pairs 702 and 704 based on the metrics calculated in stages 714A and 714B. Specifically, in this example, algorithm 700 calculates the score as SCORE = (W1*ΔC1) + (W2*ΔC2) + ... + (Wk*ΔCk), where ΔCi (as described above) is the difference indicator corresponding to Ci of the two images 702 and 704. The weights W1, W2, ... Wk can represent the importance of achieving similarity for each metric, specifically which metrics are more or less important for achieving equivalent AVI inspection performance. These weights may be application-specific to some extent. In some embodiments or scenarios, for example, pixel intensity levels may be more important than defocus, and therefore the weight of intensity difference may be heavier than the difference in Laplacian operator scalar values ​​(or other metrics indicating defocus).

[0081] As mentioned above Figure 6The image comparison tool 600 can cause the output unit 360 to display some or all of the calculated metrics in real time (e.g., only the difference metrics ΔC1 to ΔCk, or only the difference metrics exceeding the corresponding threshold, etc.). Alternatively or additionally, the tool 600 can cause the output unit 360 to display the score generated in stage 716. In embodiments where the score is calculated and displayed and is considered sufficient (e.g., above a predetermined threshold), the engineer or other user can determine that the simulated AVI station is sufficiently matched to the original AVI station and continue to use the simulated AVI station for troubleshooting and / or optimization.

[0082] If the score is insufficient, or if no score is presented and individual metrics seem insufficiently close, the user can analyze the displayed metrics to “slightly adjust” the simulated AVI station as needed for a better match. For example, if the metrics indicate that the simulated image 704 as a whole is darker than the original image 702 (e.g., has a low average intensity), the user might check the lighting or camera setup settings (e.g., light intensity settings, lens aperture settings, camera gain settings, camera exposure time settings, etc.), and / or move the lighting setup closer to the sample, etc.

[0083] There are multiple aspects associated with the illumination source that can affect the intensity of image pixels. Typically, in typical AVI applications, the effect can be macroscopic across the entire image. High-end industrial LEDs are commonly used in modern AVI stations. When setting up a simulated AVI station, the intensity can decrease if the LED source is not placed at the correct distance from the container. The intensity decreases with the square of the distance from the source, and therefore even a slight difference in positioning can have a detectable effect on the final image. Once its position relative to the container is set, the brightness of the LED source can still vary due to the power supply. Once all other factors are eliminated, the image comparison tool 600 can be used in real time to fine-tune the power and subsequent brightness of the LED source. Image processing techniques (such as those described above) offer a more precise and nuanced solution than the conventional method of measuring light source brightness using an illuminance meter in AVI applications.

[0084] Optical components such as lenses may include manual apertures or other parts that can affect overall image intensity (and potentially other image characteristics, such as aperture settings that affect image sharpness). These are typically manual dials or knobs on the lens, without digital feedback. In some cases, these components also lack any kind of visible gradient or scale. Telecentric lenses are commonly used in automated factory inspection stations to achieve high-fidelity images of products. These lenses contain pinhole apertures, the size of which can be manually adjusted in some models. This affects the amount of light allowed through the lens and has a characteristic impact on image sharpness. Therefore, by considering and combining both factors, users observing the indicators can appropriately set the lens characteristics associated with intensity.

[0085] A large, uniform drop in intensity across an image can also indicate incorrect filter placement across the illumination source (such as a polarizer or diffuser) or in front of the camera (such as an alternative polarizer or wavelength filter). Therefore, when a large difference in image intensity is observed, the user can try adjusting the filter placement.

[0086] As another example, users examining the intensity histograms and / or the output of the low-pass frequency filter for images 702 and 704 can determine whether there are significant local differences in intensity between images 702 and 704. If such local differences do exist, users can attempt to identify and remove the sources of these local differences by examining images 702 and 704, as well as their histograms and / or other metrics.

[0087] As another example, if the metrics indicate that the container in the analog image 704 is misaligned and / or improperly scaled relative to the container in the original image 702, the user can adjust the alignment of the container and / or the camera (e.g., angle or rotation), the distance between the container and the camera, the zoom level of the camera (e.g., lens type or digital zoom setting), etc.

[0088] As another example, if an indicator (e.g., the difference in output of a scalar Laplacian operator) shows that one of images 702 and 704 is out of focus relative to the other, the user can adjust the distance between the container and the camera, the motor speed, the camera exposure time, etc. Telecentric lenses, a type frequently used in inspection applications, typically have a very short depth of field, meaning that small errors in the relative placement of the lens and the container can cause image blur. Therefore, even small distance differences can have a significant impact on defocus. Furthermore, as discussed above, some of these lenses have a variable pinhole aperture, which, if improperly set, can also cause image blur. Therefore, if the indicator indicates poor defocus, the user can also adjust that aperture.

[0089] Other metrics can guide users to adjust these and / or other aspects of the simulated AVI station, for example, to achieve better matching of stray reflections in the image, key objects present in the image, image dynamic range, image bleaching, image contrast, etc.

[0090] In some embodiments, as described above, the image comparison tool 600 generates one or more suggestions based on these metrics. Thus, for example, the image comparison tool 600 may cause the output unit 360 to display (and / or generate computer voice messages to describe) any of the remedial techniques discussed above (e.g., increasing or decreasing the distance between the camera and the container, and possibly increasing or decreasing the lens aperture, etc., if the metrics reflecting the intensity and / or defocus difference of images 702, 704 are above a threshold level).

[0091] Figure 8 This is a flowchart of an example method 800 for replicating the performance of an AVI station (e.g., AVI station 310-i in a troubleshooting scenario, or a lab-based setup for development, etc.). In method 800, at block 802, a simulated AVI station (e.g., simulated AVI station 304 in a troubleshooting scenario, or AVI station 310-i, etc.) is constructed to perform one or more AVI functions of the original AVI station. Block 802 can be performed manually by reverse engineering the original AVI station. However, in some embodiments, block 802 includes using software to assist the reverse engineering process (e.g., by interpreting CAD files or the output of a 3D scanner). For example, block 802 may include the first iteration of phase 122.

[0092] At box 804, one or more container images (i.e., images of the container at the appropriate imaging location within the original AVI station) are captured by the imaging system of the original AVI station (e.g., a single camera). At box 806, one or more additional container images are captured by the imaging system of the analog AVI station (e.g., a single camera of the same type). For example, boxes 804 and 806 may be similar to stages 112 and 124 of process 100, respectively.

[0093] Subsequently, at box 808, one or more differences are identified between the container(s) captured by the original AVI station and the container(s) captured by the simulated AVI station. Box 808 can be executed by a processing unit (e.g., processing unit 342) that performs an image comparison tool (e.g., tool 350). For example, the image comparison tool can generate one or more indicators reflecting these differences at box 808. For example, box 808 may include stages 714A and 714B of algorithm 700, and difference indicators (e.g., Figure 7 The subsequent generation of indices ΔC1 to ΔCk.

[0094] At box 810, visual indicators of the differences identified at box 808 are generated to assist the user in modifying the simulated AVI station. These visual indicators may include one or more of the metrics (e.g., difference metrics) calculated at box 808 and / or one or more suggestions based on these metrics (e.g., as combined above). Figure 6 and Figure 7 (As discussed). For example, box 810 may be executed by the same processing unit that executes box 808, and may include causing an output unit (e.g., output unit 360) to present visual indications (i.e., (a plurality of) indicators and / or suggestions).

[0095] At box 812, the simulated AVI station is modified based on the visual instructions generated at box 810. Box 812 can be performed entirely by the user (i.e., manually) or can be performed at least partially automatically (e.g., by adjusting the digital settings of the camera or lighting fixtures of the simulated AVI station via computing system 340). For example, box 812 may include a second (or subsequent) iteration of phase 122 of process 100.

[0096] Although systems, methods, apparatuses, and components thereof have been described with reference to exemplary embodiments, they are not limited thereto. The detailed description is to be construed as exemplary only and does not describe every possible embodiment of the invention, as describing every possible embodiment would be neither possible nor practical. Many alternative embodiments may be implemented using current technology or technology developed after the date of this patent application, and these embodiments still fall within the scope of the claims defining the invention.

[0097] Those skilled in the art will recognize that various modifications, alterations, and combinations can be made to the embodiments described above without departing from the scope of the invention, and such modifications, alterations, and combinations can be considered to be within the scope of the inventive concept.

Claims

1. A method for replicating the performance of an Automated Visual Inspection (AVI) station, the method comprising: A simulated AVI station is provided to perform one or more AVI functions of the AVI station, wherein the simulated AVI station includes a simulated irradiation system, a simulated imaging system, and simulated sample positioning hardware configured to hold and / or support a container holding a sample. One or more images of the container are captured by the imaging system of the AVI station while the container is illuminated by the AVI station's illumination system; Simultaneously, the simulated imaging system captures one or more additional images of the container while the container is being illuminated by the simulated illumination system. One or more processors identify one or more differences between the one or more additional container images and the one or more container images; The one or more processors generate visual indications for: (i) the one or more differences, and / or (ii) one or more suggestions for modifying the simulated AVI station; and The simulated AVI station is modified at least by altering the simulated irradiation system, the simulated imaging system, and / or the simulated sample positioning hardware, based on the visual indication.

2. The method as described in claim 1, wherein, This simulated AVI station is included in commercial production line equipment.

3. The method as described in claim 1, wherein, Modifications to this simulated AVI site include: Modify the spatial arrangement of at least one illumination device of the simulated irradiation system, at least one imaging device of the simulated imaging system, and / or the simulated sample positioning hardware; and / or Modify at least one illumination device of the simulated irradiation system, at least one imaging device of the simulated imaging system, and / or the hardware components of the simulated sample positioning hardware.

4. The method of claim 2, further comprising: AVI software implementing the inspection algorithm is installed on the computer system of the simulated AVI station, and the inspection algorithm is also implemented by the AVI device; as well as After modifying the simulated AVI site While the test container is being irradiated by the simulated irradiation system, one or more new images of the container are captured by the simulated imaging system. The test container and / or the sample within the test container are determined by processing the one or more new container images according to the inspection algorithm.

5. The method of claim 4, further comprising: Troubleshooting or optimization of the simulated AVI station is based at least in part on one or more of the identified characteristics, wherein troubleshooting or optimization of the simulated AVI station includes initial modifications to previously unadjusted components of the simulated irradiation system, the simulated imaging system, the simulated sample positioning hardware and / or the AVI software, and / or iterative additional modifications to previously modified components of the simulated irradiation system, the simulated imaging system, the simulated sample positioning hardware and / or the AVI software.

6. The method of claim 5, further comprising: The AVI station is modified according to any modifications or additional modifications made to the simulated irradiation system, the simulated imaging system, the simulated sample positioning hardware, and / or the AVI software.

7. The method of claim 1, wherein: The one or more container images include a first plurality of container images; The one or more additional container images include a second or more container images; and Identifying one or more differences between the one or more additional container images and the one or more container images includes identifying one or more differences between a first composite image obtained from the first plurality of container images and a second composite image obtained from the second plurality of container images.

8. The method of claim 1, wherein, Identifying one or more differences between the one or more additional container images and the one or more container images includes identifying differences regarding the following aspects: Container alignment; Defocusing; strength; Intensity change; Motion blur; and / or Imaging sensor noise.

9. The method of claim 1, wherein, Identifying one or more differences between the one or more additional container images and the one or more container images includes calculating (i) the Fast Fourier Transform (FFT) of the one or more container images or the composite image derived therefrom, and (ii) the FFT of the one or more additional container images or the composite image derived therefrom.

10. The method of claim 1, wherein, Identifying one or more differences between the one or more additional container images and the one or more container images includes generating (i) a histogram of the pixel intensity levels of the one or more container images or the composite image derived therefrom, and (ii) a histogram of the pixel intensity levels of the one or more additional container images or the composite image derived therefrom.

11. The method of claim 1, wherein, Identifying one or more differences between the one or more additional container images and the one or more container images includes calculating (i) the Laplacian operator of the one or more container images or the composite image derived therefrom, and (ii) the Laplacian operator of the one or more additional container images or the composite image derived therefrom.

12. The method of claim 1, wherein, While capturing the one or more additional container images, perform in real time (i) identify the one or more differences and (ii) generate the visual indication.

13. The method of claim 1, further comprising: The processor generates one or more suggestions for modifying the simulated AVI station based on the one or more differences. Generating the visual instruction includes generating the visual instruction for the one or more suggestions.

14. The method of claim 13, wherein, Generating one or more recommendations includes generating: Recommendations for modifying one or more configurable settings of at least one imaging device of the analog imaging system; Recommendations for modifying one or more configurable settings of at least one lighting device of the simulated illumination system; Recommendations for modifying the position of at least one imaging device of the analog imaging system; and / or Recommendations for modifying the position of at least one lighting device in the simulated illumination system.

15. The method of claim 13, wherein: The simulated sample positioning hardware is configured to move the container according to a movement curve; and Generating one or more suggestions includes generating suggestions for modifying one or more properties of the moving curve.

16. The method of claim 1, further comprising: The one or more processors receive one or more operating parameters of the imaging system and one or more operating parameters of the analog imaging system; The one or more processors compare one or more operating parameters of the imaging system with one or more operating parameters of the analog imaging system; as well as The processor generates additional visual indications of at least one operating parameter that differs between the imaging system and the analog imaging system.

17. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the following operations: Receive one or more container images captured by the imaging system of an Automated Visual Inspection (AVI) station; Receive one or more additional container images captured by an analog imaging system of an analog AVI station included in commercial production line equipment; Identify one or more differences between the one or more additional container images and the one or more container images; as well as Generate visual indications for: (i) the one or more differences, and / or (ii) one or more suggestions for modifying the simulated AVI station.

18. The non-transitory computer-readable medium of claim 17, wherein, These instructions enable the one or more processors to recognize one or more differences between the one or more additional container images and the one or more container images in terms of: Container alignment; Defocusing; strength; Intensity change; Motion blur; and / or Imaging sensor noise.

19. The non-transitory computer-readable medium of claim 17, wherein, These instructions enable the one or more processors to identify one or more differences between the one or more additional container images and the one or more container images, at least in the following ways: Calculate the Fast Fourier Transform (FFT) of the one or more container images or the composite image derived from them; and Calculate the FFT of the one or more attached container images or the composite image derived from them.

20. The non-transitory computer-readable medium of claim 17, wherein, These instructions enable the one or more processors to identify one or more differences between the one or more additional container images and the one or more container images, at least in the following ways: Generate a histogram of pixel intensity levels for the one or more container images or composite images derived therefrom; and Generate a histogram of pixel intensity levels for the one or more additional container images or the composite image derived from them.

21. The non-transitory computer-readable medium of claim 17, wherein, These instructions enable the one or more processors to identify one or more differences between the one or more additional container images and the one or more container images, at least in the following ways: Calculate the Laplacian operator for the one or more container images or the composite image derived therefrom; and Calculate the Laplacian operator for the one or more additional container images or the composite image derived from them.

22. The non-transitory computer-readable medium of claim 17, wherein, These instructions enable the one or more processors to (i) identify the one or more differences and (ii) generate the visual indication in real time upon receiving the one or more additional container images.

23. The non-transitory computer-readable medium of claim 17, wherein: These instructions cause the one or more processors to generate one or more suggestions for modifying the simulated AVI station based on the one or more differences; and The visual indicator refers to one or more recommendations.

24. The non-transitory computer-readable medium of claim 23, wherein, The one or more recommendations include: Recommendations for modifying one or more configurable settings of at least one imaging device of the analog imaging system; Recommendations for modifying one or more configurable settings of at least one lighting device of the simulated illumination system of the simulated AVI station; Recommendations for modifying the position of at least one imaging device of the analog imaging system; and / or Recommendations for modifying the position of at least one lighting device in the simulated illumination system.

25. A system comprising: Automated Vision Inspection (AVI) station, which includes Imaging system, Irradiation system, and Sample positioning hardware, configured to hold and / or support a container holding a sample; A simulated AVI station that performs one or more AVI functions and is included in commercial production line equipment, the simulated AVI station including... Simulated irradiation system Analog imaging systems, and Simulated sample positioning hardware, which is configured to hold and / or support a container holding a sample; as well as A computing system configured to receive one or more container images captured by the imaging system of the AVI station; Receive one or more additional container images captured by the analog imaging system of the analog AVI station; Identify one or more differences between the one or more additional container images and the one or more container images; as well as Generate visual indications for: (i) the one or more differences, and / or (ii) one or more suggestions for modifying the simulated AVI station.

26. The system of claim 25, wherein, The computing system is configured to identify one or more differences between the one or more additional container images and the one or more container images regarding the following: Container alignment; Defocusing; strength; Intensity change; Motion blur; and / or Imaging sensor noise.

27. The system of claim 25, wherein, The computing system is configured to identify one or more differences between the one or more additional container images and the one or more container images at least in the following ways: Calculate the Fast Fourier Transform (FFT) of the one or more container images or the composite image derived from them; and Calculate the FFT of the one or more attached container images or the composite image derived from them.

28. The system of claim 25, wherein, The computing system is configured to identify one or more differences between the one or more additional container images and the one or more container images at least in the following ways: Generate a histogram of pixel intensity levels for the one or more container images or composite images derived therefrom; and Generate a histogram of pixel intensity levels for the one or more additional container images or the composite image derived from them.

29. The system of claim 25, wherein, The computing system is configured to identify one or more differences between the one or more additional container images and the one or more container images at least in the following ways: High-pass filtering is applied to the one or more container images or composite images derived therefrom; as well as High-pass filtering is applied to the one or more additional container images or the composite image derived from them.

30. The system of claim 29, wherein, The computing system is configured to identify one or more differences between the one or more additional container images and the one or more container images at least in the following ways: Calculate the Laplacian operator for the one or more container images or the composite image derived therefrom; and Calculate the Laplacian operator for the one or more additional container images or the composite image derived from them.

31. The system of claim 25, wherein, The computing system is configured to (i) identify the one or more differences and (ii) generate the visual indication in real time upon receiving the one or more additional container images.

32. The system of claim 25, wherein: The computing system is configured to generate one or more suggestions for modifying the simulated AVI station based on the one or more differences; and The visual indicator refers to one or more recommendations.

33. The system of claim 25, wherein, The one or more recommendations include: Recommendations for modifying one or more configurable settings of at least one imaging device of the analog imaging system; Recommendations for modifying one or more configurable settings of at least one lighting device of the simulated illumination system of the simulated AVI station; Recommendations for modifying the position of at least one imaging device of the analog imaging system; and / or Recommendations for modifying the position of at least one lighting device in the simulated illumination system.

Images