Sheet illumination for particle detection in pharmaceutical product containers

Abstract

In a method for imaging a container containing a sample, the container is illuminated with a laser sheet that illuminates the container in a first direction corresponding to a first axis. A plane of the laser sheet is defined by the first axis and a second axis orthogonal to the first axis. The method also includes capturing an image of the container by a camera having an imaging axis substantially orthogonal to at least the first axis. The method further includes analyzing, by one or more processors, the image of the container to detect particulates within and / or on the outer surface of the container.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a divisional application of Application No.: 201980083107.8 Invention Name: Sheet Lighting for Particle Detection in Pharmaceutical Product Containers. This application claims priority to U.S. Provisional Patent Application No. 62 / 780,542 filed on December 17, 2018, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application relates generally to particle detection technology, and more particularly to particle detection technology capable of distinguishing particles within a container (eg, a syringe, a vial, etc.) from particles on an external surface of the container. Background Art

[0004] Foreign matter in drug product containers can pose a serious health and safety risk to patients, particularly for injectable drug products. Although automated visual inspection equipment can detect particles in containers containing liquid products with acceptable accuracy in some cases, there can be a large number of false positives, which are caused, for example, by small particles and spots on the exterior of the container wall, defects (e.g., cracks) on the interior or large portions of the container wall, and / or small bubbles inside the container wall.

[0005] A conventional imaging system 100 is shown in FIG. 1 . As shown in FIG. 1 , a container 102 filled with a sample (e.g., a liquid drug product) is illuminated with two angled beams of light 104A and 104B, which are positioned generally opposite a camera 106 on the other side of the container 102. The lights 104A and 104B are directional, with most of the emitted light propagating in a direction orthogonal to the flat surface shown in FIG. 1 (i.e., as indicated by the arrows in FIG. 1 ). This requires that each beam of light 104A, 104B is large enough to evenly illuminate the entire container. Since most of the emitted light is not directed toward the lens of the camera 106, if any particles are present, the resulting image is a bright particle against a dark background. FIG. 2 depicts another conventional imaging system 200, which is typically used for larger opaque particles (over 500um) or fibers that may adhere to the inner wall of the container. As shown in FIG. 2 , the container 202 is illuminated by a backlight 204 opposite the camera 206. In this case, the camera 206 images the shadow cast by the particle, and the particle appears as a dark object against a relatively bright background.

[0006] The problem with these two conventional illumination methods is that the entire container is flooded with light, so that particles and surface spots inside and outside the container are illuminated. As a result, it may be difficult to distinguish particles inside the container from particles outside the container. Since particles outside the container may be irrelevant to the quality control procedure, this difficulty increases the risk of false positives. For back-angled light devices such as imaging system 100, a technique known as "image subtraction" or "minimum intensity projection" (MIP) is usually used to distinguish smaller particles (about 100um to 500um) inside the container from particles outside the container. This technique involves rotating the container at a high speed (about 600RPM to 5000RPM) around its central axis, suddenly stopping the rotation, acquiring a series of images of the stopped container at intervals of about 10ms to 50ms, and then subtracting subsequent images so that only those objects that move between images will appear in the resulting difference image. This effectively eliminates small particles and surface spots that may be outside the container, while highlighting objects suspended in the liquid in the container and carried by the momentum of the liquid. However, this technique may not be suitable for highly viscous pharmaceutical products because there is little or no movement of the fluid and particles after the container stops rotating. Backlighting devices such as imaging system 200 also have disadvantages because they may "bleach" small or opaque particles and generally rely on particles inside the container being large enough to be distinguished from smaller particles that are typically on the outer surface of the container.

[0007] Some manufacturers of automated inspection equipment for pharmaceutical products have proposed and implemented technologies that attempt to address these problems. For example, U.S. Pat. No. 8,036,444 (Nielsen), entitled "Method and System for Irradiating and Inspecting Liquid-Carrying Containers," describes an imaging system in which two line scan cameras generate a flattened image of a rotating container. One camera is aligned with the central axis of the container, while the other camera is offset from the central axis. The technology exploits the following basic principle: when the container rotates, particles outside the container move a longer distance in the horizontal direction (i.e., in a direction orthogonal to the central axis of the container) than particles inside the container. After multiple images have been acquired at different rotations, the images from the two line scan cameras are compared, and the distance between the particles is calculated. In some cases, this distance can be used to distinguish between particles inside and outside the container.

[0008] As another example, European Patent No. 3,062,292 (Kwoka), entitled "Inspection Method and Inspection Device for Monitoring Production Processes," describes an imaging system using a single-sided scanning camera. When a particle is detected at a certain position along the central axis of a container, the position is digitally shifted to the point where the particle would be if the particle was outside the container and the container was precisely rotated about its central axis by a preset angle (approximately 45°). The container is then rotated and a new image is taken. If the particle is actually outside the container, the image should overlap with the digitally shifted image. Conversely, if the particle is inside the container, the particle will be offset by a certain amount from the digitally shifted particle.

[0009] Although the techniques of U.S. Pat. No. 8,036,444 (Nielsen) and European Patent No. 3,062,292 (Kwoka) may improve upon the conventional image subtraction methods described above in some respects, both methods have their own significant disadvantages. One difficulty is that in both methods, any slight vibration of the container between images may result in significant errors. In addition, if a particle inside the container "slips" when the container is rotated, the particle may not be in the expected position where it will be correctly detected. Furthermore, limitations in the spatial resolution of the imager may render the technique inadequate for distinguishing small spots inside the glass, which can be a major source of false positives.

[0010] Further, while promising 3D imaging techniques are being developed and provide the ability to determine particle size and morphology from images, they are computationally expensive and may not work at typical production line rates (e.g., 300 to 600 containers per minute). Thus, there remains a need for improved methods to detect particles within liquid-filled pharmaceutical product containers, particularly (but not limited to) containers containing highly viscous samples / products. Summary of the invention

[0011] Embodiments described herein relate to systems and methods that improve upon conventional automated visual inspection techniques. In particular, an imaging system illuminates a container with a relatively thin laser sheet, wherein the laser sheet illuminates the container from a direction substantially orthogonal to the imaging axis of the camera. For example, the laser sheet may pass through the central axis of the container. Under this illumination configuration, particles are visible in the resulting image only if they are within the thickness of the laser sheet (or are visible at a certain intensity level, etc.). Thus, by using the laser sheet and the orthogonal orientation of the camera, it is easily determined whether a particle seen in an image is inside or outside the container: any imaged particle outside the container will be outside the container walls in the image, and any imaged particle inside the container will be between the container walls in the image.

[0012] In some embodiments, the container is rotated multiple times about its central axis and the container is imaged after each rotation while the laser sheet is still applied so as to examine the entire volume of the drug product (or other sample) within the container. Although many rotations may be required, the image processing and computational load can be very light and the likelihood of misclassification can be very low. In addition, the technique has the unexpected benefit that a portion of the laser light is scattered within the sample, or refracted at the container / sample (e.g., glass / liquid) interface, and travels around the interior circumference of the container (e.g., if the container is cylindrical). This scattered or refracted light can illuminate bubbles located anywhere inside the container, even if they do not intersect the laser sheet. This phenomenon is caused by the large difference in refractive index at the surface (which results in a large amount of reflection and refraction), and can be used to more accurately distinguish bubbles from particles (e.g., debris, protein aggregates, etc.) than other methods that rely solely on the different morphologies of bubbles and particles. The ability to better distinguish bubbles (usually benign) from other particles can be important because bubbles are often a significant source of misclassification (e.g., when using conventional image subtraction methods).

[0013] Other more complex arrangements may be used to construct the laser sheet technique. For example, two laser sheets (both orthogonal to a single camera) may be used that are 180 degrees apart from each other to compensate for the fact that the light scattering of the laser sheet is different at the location where the laser sheet enters the container than at the location where the laser sheet leaves the container. This may cut the number of required rotations / imaging in half. As another example, one or more additional laser sheets may be applied at an angle oblique to the first laser sheet to better image a specific area of ​​the container (e.g., a shoulder or stopper area). As yet another example, the imaging system may include a first laser source that generates a laser sheet of one color (e.g., red) and a second laser source that generates a laser sheet of another color (e.g., blue), wherein there is an angular offset between the two laser sheets relative to the central axis of the container. Images may then be captured simultaneously with two cameras (each tuned to a different of the two colors). Alternatively, a single camera (e.g., with one or more mirrors, prisms, and / or other optical components) can be used to capture images that preserve the visual information provided by the illumination of each of the different colored laser sheets (e.g., using a camera that implements a Bayer color filter, or using a camera that includes optics and filters to map the different colors of the two laser sheets to different parts of the camera sensor). Whether one or two cameras are used, this approach can reduce the number of required rotations / imaging by half (or by one-third if three different colored laser sheets are used). As yet another example, an imaging system can include a laser source that generates a laser sheet of one color (e.g., red), and an illumination source that generates light of another color (e.g., blue) that illuminates substantially the entire volume of the container. By using cameras tuned to different colors, particles inside and outside the container can be distinguished in one plane (within the laser sheet), and a snapshot of the entire volume can be obtained simultaneously (using the other illumination source).

[0014] The techniques described above and elsewhere herein may provide a number of advantages, such as enabling automated detection of fibers and other particles in highly viscous products, allowing accurate automated detection of small particles adhering to the inner wall of a container, improving discrimination of bubbles from particles inside a container, avoiding false detections or other problems due to small spots on the inner wall of a container, reducing the risk of rejection due to particles that are not actually inside the container or due to bubbles (i.e., reducing false positives that may result in the discarding of an entire batch of drug product), reducing the need to perform expensive manual inspections to avoid false positives, and / or reducing patient risk. Furthermore, these techniques may be implemented by retrofitting current automated inspection equipment with minimal hardware changes (e.g., by simply adding one or more laser sources). BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Those skilled in the art will appreciate that the drawings described herein are included for illustrative purposes and do not limit the present disclosure. The drawings are not necessarily drawn to scale, but rather focus on illustrating the principles of the present disclosure. It should be understood that in some cases, different aspects of the described embodiments may be enlarged or magnified to aid in understanding the described embodiments. In the drawings, similar reference numerals generally refer to components that are similar in function and / or structure throughout the drawings.

[0016] Figures 1 and 2 depict conventional imaging systems for particle detection.

[0017] Figure 3A and Figure 3B Depicted are different perspective views of a first embodiment of an example imaging system operating according to the principles described herein.

[0018] Figure 4 Describes the Figure 3A and Figure 3B Example images of a container illuminated by a similar imaging system.

[0019] Figure 5 A second embodiment of an example imaging system operating according to the principles described herein is depicted.

[0020] Figure 6 A third embodiment of an example imaging system operating according to the principles described herein is depicted.

[0021] Figure 7 A fourth embodiment of an example imaging system operating according to the principles described herein is depicted.

[0022] Figure 8 It can be compared with Figure 3. Figure 5 , Figure 6 or Figure 7 A simplified block diagram of an example automated inspection system for use with an imaging system.

[0023] Fig. 9 is a flow chart of an example method for imaging a container containing a sample. DETAILED DESCRIPTION

[0024] The various concepts introduced above and discussed in more detail below may be implemented in any of a variety of ways, and the concepts described are not limited to any particular implementation.For purposes of illustration, examples of implementation are provided.

[0025] exist Figure 3A and Figure 3B The first embodiment is shown in FIG. 3 , which provides different perspective views of an example imaging system 300. Specifically, Figure 3A provides an off-axis perspective, while Figure 3B Provides a top-down view. Figure 3A and Figure 3B In FIG. 1 , one or more images of a container 302 (in a holder 303) are captured by an imager 304 while the container 302 is being illuminated by a laser source 306. Figure 3A and Figure 3B Container 302 is depicted as a syringe, but it should be understood that container 302 can alternatively be any other suitable type of container and can have any suitable size and shape. For example, container 302 can alternatively be a medicine bottle, a test tube, a cartridge, etc. Container 302 can be made of glass, plastic, or any other suitable material (or material combination) that is at least partially transparent or translucent to allow the passage of light from laser source 306 and allow the light to pass through to imager 304. In operation, container 302 can hold a liquid sample. However, in some use cases, container 302 can hold a non-liquid sample, such as a freeze-dried or frozen sample.

[0026] The holder 303 may include any hardware required to maintain the container 302 in a desired position and to rotate the container 302 to allow the imager 304 to capture images from other angles. The holder 303 may be only a part of some suitable devices for positioning the container 302 in one or more desired positions and / or orientations. Depending on the requirements of the imaging system 300, the positioning device may include any suitable combination of hardware, firmware and / or software. For example, the positioning device may include only a platform (e.g., a flat base member), and the holder 303 protrudes vertically from the platform in a fixed orientation or in a manner that allows the holder 303 to rotate. However, in other embodiments, the positioning device may include automatic / robotic hardware (e.g., a mechanical arm including the holder 303 or another suitable holding device (such as a "fingers") that can grasp / pinch the container 302). In these latter embodiments, the positioning device may also include a processing unit (e.g., a microprocessor and / or an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA), etc.), and a memory storing instructions (e.g., a solid-state memory or a hard disk memory), and the processing unit may execute these instructions to grasp / hold / fix, shift and / or rotate the container 302. Other positioning devices are of course possible.

[0027] For example, the imager 304 can be a camera including one or more charge coupled device (CCD) sensors. Alternatively, the imager 304 can include one or more complementary metal oxide semiconductor (CMOS) sensors and / or any other suitable type of imaging device / sensor. The imager 304 can include, for example, a telecentric lens, or any other suitable lens (or combination of multiple lenses). In various embodiments, the imager 304 can include any suitable combination of hardware and / or software, such as an image sensor, a light stabilizer, an image buffer, a frame buffer, a frame grabber, etc. More generally, the imaging system 300 can include any suitable device for capturing one or more images of the container 302 (or another suitable container), wherein the imaging device includes the imager 304 and / or any other suitable one or more imaging devices (e.g., the imager 304 plus one or more mirrors, additional lenses, etc.).

[0028] The laser source 306 generates a laser sheet 310 that generally conforms to a plane. Although the laser sheet 310 is referred to herein as a "sheet," it should be understood that practical limitations of the laser source 306 and the medium through which the laser sheet 310 passes (i.e., air or other gas, the walls of the container 302, and the liquid or other sample within the container 302) will prevent the laser sheet 310 from forming a uniform, flat sheet. For example, the laser sheet 310 will experience some diffusion when entering and exiting the container 302. In some embodiments, the laser source 306 is a diode laser having a power of 1 mW to 5 mW, a line fan angle of 30 to 60 degrees, and a line width (thickness) of 1 mm to 1.5 mm. In one embodiment, the laser source 306 is an Edmund Optics Micro VLM Laser Diode Line Part #52-267, which has a power of 3.5 mW, a wavelength of 670 nm, and a line fan angle of 60 degrees. In another embodiment, the laser source 306 is an Edmund Optics Micro VLM Laser Diode Line Part #52-268 having a power of 1.6 mW, a wavelength of 670 nm, and a line fan angle of 30 degrees. More generally, the imaging system 300 may include any suitable device for generating the laser sheet 310, such as the laser source 306 or another suitable laser source. In some embodiments (e.g., if the container 302 is dark brown so as to block visible light), the laser source 306 uses an infrared laser to generate the laser sheet 310. As used herein, "light" does not necessarily refer to the portion of the electromagnetic spectrum visible to humans.

[0029] like Figure 3AAs shown in the exemplary embodiment of , the laser sheet 310 illuminates the container 302 in a direction 322 corresponding to (i.e., aligned with) the first axis 330 and generally coincides with a plane defined by the first axis 330 and a second orthogonal axis 332. In the depicted embodiment, the second axis 332 is parallel to the central axis 320 of the container 302. In addition, as shown in FIG. Figure 3B As more clearly shown in , in the depicted embodiment, the imaging axis 324 of the imager 304 passes through the center of the container 302 (i.e., through the central axis 320). The imaging axis 324 of the imager 304 is substantially parallel to the third axis 334, wherein the third axis 334 is orthogonal to both the first axis 330 and the second axis 332. In this context, the term "substantially" is used to reflect the fact that the alignment of the components is never absolutely perfect, and to indicate that small deviations are acceptable as long as they do not destroy the main benefits provided by the technology described herein. For example, in some use cases, the imaging axis 324 can be within five degrees of perfect orthogonality of the axes 330 and 332, or within three degrees, within two degrees, within one degree, etc.

[0030] The laser sheet 310 has a finite thickness 340 that covers a smaller range of the third axis 334. For example, the thickness 340 may represent 3 times the beam width of the laser sheet 310 along the third axis 334. Because the laser source 306 is not ideal (theoretically perfect), the thickness 340 is not exactly uniform at all points along the axis 330. However, at least at the location where the laser sheet 310 enters the container 302, the thickness 340 is substantially less than the diameter of the container 302. The thickness 340 may be set as a design parameter based on both the size (e.g., diameter) of the container 302 and the desired (or maximum acceptable, etc.) number of rotations / imaging. In particular, the thickness 340 may be set so that when the container 302 is rotated a certain number of times (to allow the imager 304 to capture images from those angles), and each rotation has a certain angular offset, all portions of the volume of the container 302 (or a large percentage thereof) will eventually be illuminated. This may also require consideration of whether the laser sheet 310 sufficiently illuminates both sides of the container 302 (i.e., both the location where the laser sheet 310 enters the container 302 and the location where the laser sheet leaves the container 302) for any given rotational position of the container 302. For example, if the container 302 is rotated / imaged 90 times to achieve full coverage, and if the laser sheet 310 sufficiently illuminates both the entry and exit sides of the container 302, then the thickness 340 may be set such that the laser sheet 310 covers 1 / 90 (or just over 1 / 90) of the circumference of the container 302, wherein half of the coverage corresponds to the location where the laser sheet 310 enters the container 302 and the other half of the coverage corresponds to the location where the laser sheet 310 leaves the container 302. On the other hand, if the laser sheet 310 does not sufficiently illuminate the exit side of the container 302, then the thickness 340 may still be set such that the laser sheet 310 covers approximately 1 / 90 of the circumference of the container 302, but now the coverage all occurs at the location where the laser sheet 310 enters the container 302. Thus, for example, if the diameter of the container 302 is 100 mm, and the laser sheet 310 fully illuminates both sides of the container, the thickness 340 can be set to about 2 mm: the coverage of the laser sheet 310 is 100 mm*π / (90 rotations)=3.49 mm / each rotation (half of the coverage occurs on each side of the container 302, so that the thickness 340 can be 3.49 mm / 2=1.74 mm, allowing an additional 0.26 mm of thickness to ensure complete coverage). Conversely, if the laser sheet 310 does not fully illuminate the exit side of the container 302, the thickness 340 can be set to about 4 mm (or alternatively, the number of rotations is increased from 90 to 180).

[0031] The lower limit of thickness 340 may be limited by the need to provide complete coverage / illumination, while the upper limit of thickness 340 may be limited by the need to avoid illuminating too much of the wall of container 302 for any given rotation / image. In particular, if laser sheet 310 illuminates any particles that are outside of container 302 but appear between the outermost boundaries of container 302 (from the perspective of imager 304), the improved discrimination provided by imaging system 300 begins to diminish. In various embodiments, thickness 340 is greater than zero but less than 1 mm, less than 2 mm, less than 3 mm, less than 4 mm, less than 5 mm, etc. When stated as a range, in various embodiments, thickness 340 may be somewhere between 1 mm and 3 mm, somewhere between 1 mm and 5 mm, somewhere between 0.5 mm and 5 mm, etc.

[0032] In some embodiments, the distance of the laser source 306 from the container 302 and the beam angle of the laser sheet 310 along the second axis 332 may be fixed so that the laser sheet 310 illuminates the entire cross-section of the container 302 in the plane defined by the axes 330, 332. However, in other embodiments, the laser sheet 310 illuminates only a smaller cross-section of the container 302 (e.g., not including the second axis 332). Figure 3A In some embodiments, the laser sheet 310 may be a laser beam having a plurality of projections, a plurality of projections, and / or a plurality of projections. ...

[0033] In some embodiments, the imaging system 300 is configured to Figure 3A and Figure 3B 334. As another example, the imaging axis 324 of the imager 304 can be angled such that the imaging axis is slightly raised upward or tilted slightly downward from the plane defined by the axes 330, 334. As yet another example, the direction 322 of the laser sheet 310 can be angled such that the direction is raised upward or tilted slightly downward from the plane defined by the axes 330, 334 (e.g., to better illuminate a shoulder or stopper area, etc.). As yet another example, the imaging system 300 can include one or more additional imagers and / or illumination sources (e.g., as described below with reference to Figures 5 to 7 discussed).

[0034] Figure 4 Describes the Figure 3A and Figure 3B4. Example image 400 of a container illuminated by an imaging system 300 of FIG. 4. For example, the imaged container may be container 302 when illuminated by laser sheet 310. In particular, Figure 4 This corresponds to a situation in which the laser sheet illuminates the container from the left side of the area depicted in image 400, and an embodiment in which the laser sheet does not adequately illuminate the exit side of the container (ie, the right side of the area depicted in image 400).

[0035] Because the thickness of the laser sheet (e.g. Figure 3A and 3B The laser sheet 400 (thickness 340) illuminates only a small "slice" of the container at any one time / after any one rotation, so any particles on the outside of the container (e.g., dust or fibers) will only be illuminated (at least at a level sufficient for clear imaging) along the portion of the container wall that faces the laser sheet source (e.g., where the curved container wall has a surface normal vector that is substantially orthogonal to the imaging axis). Thus, any particles on the outside of the container wall will only be visible (or clearly visible) if the image shows that these particles are clearly outside the container. In the example image 400, this means that particles on the outside of the container will only appear immediately to the left of (and in contact with) the container wall. In embodiments where the laser sheet also substantially illuminates the exiting (right) side of the container, the image may also show that the external particles are immediately to the right of (and in contact with) the container. However, any external particles on other portions of the container wall (i.e., on a portion of the wall closer to the center of the image 400) will not be illuminated, or will at best be illuminated very weakly by scattered / refracted light. Thus, in image 400, any particles shown between the leftmost and rightmost boundaries of the container will be clearly located inside the container (ie, within the portion / slice of the sample illuminated by the laser sheet as it passes through the container interior).

[0036] like Figure 4 As shown, the laser sheet illuminates many smaller dust-like particles on the left side of the container exterior, and large particles (fibers) just inside the left wall of the container. Small particles can also be seen inside the container just to the right of the fibers in image 400. As used herein, in some use cases, the term "particle" refers to any object that is small relative to the container and solids (e.g., fibers or other debris, or protein aggregates, etc.), or may be a microemulsion.

[0037] A second benefit of imaging with the laser sheet is that the bubbles can be illuminated wherever they are within the container, even if they do not intersect the laser sheet. This is caused by the high reflectivity of the bubbles and by some of the laser light being scattered and / or refracted at the container / sample interface and / or due to interactions with objects within the sample. Figure 4. As a result, bubbles can be more easily distinguished from particles. For example, the automated image analysis / processing may initially count an object appearing between the container walls as a "candidate" particle, but if other images at other rotation points continue to show the object at a location that would be expected based on the initial object position and the known rotation angle, then determine that the object is a bubble and not an actual particle. Conversely, if the object does not reappear at the corresponding location in other images at other rotations, then the automated image analysis / processing may identify the object (a candidate particle) as an actual particle.

[0038] Figure 5 An alternative embodiment is depicted in which imaging system 500 uses imager 504, first laser source 506A and second laser source 506B to image container 502, with laser sources 506A and 506B facing directions that differ by 180 degrees. Figure 3A and Figure 3B , container 502 can be similar to container 302, imager 504 can be similar to imager 304, and each of laser sources 506A and 506B can be similar to laser source 306. Laser sources 506A and 506B generate laser sheets 510A and 510B, respectively, each of which can be similar to Figure 3A and Figure 3B However, in imaging system 500, laser sheet 510A illuminates container 502 in direction 522A, and laser sheet 510B illuminates container 502 in an opposite direction 522B. Imaging axis 524 of imager 504 is substantially orthogonal to directions 522A and 522B.

[0039] By using a second opposing laser source, imaging system 500 can better illuminate both sides of container 502 (i.e., the "left side" and the "right side" from the perspective of imager 504) after each rotation of container 502, thereby reducing the number of required rotations and imaging times by half. Alternatively, if the number of rotations / imaging times is not reduced, the second laser source can allow each of laser sheets 510A and 510B to have approximately half the thickness (relative to thickness 340 of laser sheet 310), which can help ensure that illuminated particles outside of container 502 do not appear to be just inside the wall of container 502 (e.g., at Figure 4 near the leftmost edge and the rightmost edge of the container 400, but in between).

[0040] Figure 6 Another alternative embodiment is depicted in which the imaging system 600 uses a first imager 604A, a second imager 604B, a first laser source 606A, and a second laser source 606B to image the container 602. For example, referring to Figure 3A and Figure 3B , container 602 can be similar to container 302, each of imagers 604A and 604B can be similar to imager 304, and each of laser sources 606A and 606B can be similar to laser source 306. Figure 6 In an embodiment of the present invention, laser sheet 610A generated by laser source 606A (e.g., similar to laser sheet 310) illuminates container 602 in a first direction 622A, and laser sheet 610B generated by laser source 606B (e.g., also similar to laser sheet 310) illuminates container 602 in a second direction 622B that is neither parallel nor orthogonal to first direction 622A. For example, there may be an angular displacement of 135 degrees (or 150 degrees, 120 degrees, 60 degrees, 30 degrees, etc.) between direction 622A and direction 622B. Imaging axis 624A of imager 604A is substantially orthogonal to first direction 622A, and imaging axis 624B of imager 604B is substantially orthogonal to second direction 622B.

[0041] Laser sources 606A and 606B can generate light of different wavelengths / colors. For example, laser sheet 610A can be red, while laser sheet 610B can be blue or green, etc. In addition, the filters of imagers 604A and 604B can pass only the color of the corresponding laser source (e.g., imager 604A can be configured to image red light but not blue light, and imager 604B can be configured to image blue light but not red light). By utilizing different colors, imaging system 600 allows for simultaneous imaging by imagers 604A and 604B, which can have one or more advantages. For example, imaging two "slices" of a sample at any one time can reduce the number of rotations of container 602 required by half relative to the case where a single laser sheet and imager is used, or allow the thickness of the laser sheet to be reduced, as described above in conjunction with Figure 5 discussed.

[0042] In an alternative embodiment, imaging system 600 includes imager 604A but omits imager 604B. In such an embodiment, imaging system 600 includes suitable optics (e.g., one or more mirrors, prisms, and / or other optical components) such that the optical path of imager 604A has both a first component aligned with direction 624A and a second component aligned with direction 624B. Imager 604A may include, for example, a Bayer color filter (e.g., a commonly used color CCD or CMOS chip) to capture and clearly preserve the visual information provided by the illumination from each laser sheet 610A, 610B. That is, for each rotation of container 602, a single imager 604A may capture a composite image that includes information sufficient to recreate a first image corresponding to the color of laser sheet 610A (e.g., red) and a second image corresponding to the color of laser sheet 610B (e.g., green). Alternatively, imager 604A may include a camera with optics and filters suitable for mapping visual information corresponding to different colors to different portions of the camera sensor.

[0043] Figure 7 Yet another alternative embodiment is depicted in which imaging system 700 uses a first imager 704A, a second imager 704B, a laser source 706 that generates a laser sheet 710, and an additional illumination source 712 to image container 702. For example, referring to Figure 3A and Figure 3B , container 702 can be similar to container 302, one or both of imagers 704A and 704B can be similar to imager 304, and laser source 706 can be similar to laser source 306. However, illumination source 712 may not produce a laser sheet, and in fact may not produce laser light at all. For example, illumination source 712 may include one or more light emitting diodes (LEDs) and / or another suitable light source that illuminates substantially the entire volume of container 702 at one time. Laser sheet 710 (e.g., similar to laser sheet 310) illuminates container 702 in direction 722, imaging axis 724A of imager 704A is substantially orthogonal to direction 722, and imaging axis 724B of imager 704B is neither parallel to nor orthogonal to direction 722. Although in Figure 7 704B), it should be understood that in some embodiments, the illumination source 712 can be offset from the imaging axis 724B. For example, the illumination source 712 can include one or more light sources that are positioned above and / or below the imaging axis 724B and angled downward and / or upward to illuminate the container 702 without providing direct backlighting.

[0044] Laser source 706 and illumination source 712 generate light of different wavelengths / colors. For example, laser sheet 710 can be red, and the light generated by illumination source 712 can be blue. In addition, the filters of imagers 704A and 704B can be configured to pass the color of the corresponding illumination source (e.g., imager 704A can be configured to image red light but not blue light, and imager 704B can be configured to image blue light but not red light). By utilizing different colors, imaging system 700 allows for simultaneous imaging by imagers 704A and 704B, which can have one or more advantages. For example, the image generated by imager 704B can be used to identify particles anywhere in or on container 702 for motion tracking, while the image generated by imager 704A can be used to determine which of those particles are outside of container 702.

[0045] In an alternative embodiment, imaging system 700 includes imager 704A, but omits imager 704B. Figure 6 Similar to the devices discussed, for example, imager 704A can implement a Bayer color filter (or an optical device / filter that maps different colors to different areas of the camera sensor), and imaging system 700 can include appropriate additional optical components (e.g., (multiple) mirrors and / or (multiple) prisms) to provide imager 704A with optical path components along both directions 724A and 724B. In this way, imager 704A can capture images, each of which preserves visual information provided by illumination from laser source 706 and illumination from light source 712.

[0046] Figure 8 3, Figure 5 , Figure 6 or Figure 7 The automated inspection system 800 includes a computer system 802 that receives images from an imager 804 (e.g., similar to the imager 304). The imager 804 generates one or more images of a container holding a sample while the container and the sample are illuminated by a laser sheet as described in any of the various embodiments above.

[0047] Computer system 802 may be a general purpose computer that is specifically programmed to perform the operations discussed herein, or may be a dedicated computing device (e.g., part of an imaging unit that includes imager 804). Figure 8As shown, computer system 802 includes processing unit 810 and memory unit 812. However, in some embodiments, computer system 802 includes two or more computers co-located or remote from each other. In these distributed embodiments, the operations described herein related to processing unit 810 and / or memory unit 812 can be divided between multiple processing units and / or memory units, respectively.

[0048] The processing unit 810 constitutes a processing device for analyzing images of containers to detect particles inside and / or on the outer surface of those containers. The processing unit 810 includes one or more processors, each of which can be a programmable microprocessor that executes software instructions stored in the memory unit 812 to perform some or all of the functions of the computer system 802 as described herein. For example, the processing unit 810 may include one or more graphics processing units (GPUs) and / or one or more central processing units (CPUs). Alternatively or additionally, some processors in the processing unit 810 may be other types of processors (e.g., ASICs, FPGAs, etc.), and a certain function of the computer system 802 as described herein may be implemented in hardware instead. The memory unit 812 may include one or more volatile and / or non-volatile memories. Any suitable one or more memory types may be included, such as read-only memory (ROM), random access memory (RAM), flash memory, solid-state drive (SSD), hard disk drive (HDD), etc. The memory unit 812 may store instructions for one or more software applications, data received / used by these applications, and data output / generated by these applications together.

[0049] One such software application stored in memory unit 812 is a particle detection application 814 which, when executed by processing unit 810, processes images generated by imager 804 (and possibly also images generated by one or more other imagers such as Figure 6 Imager 604B or Figure 7 The particle detection application 814 may also perform other operations, such as scoring a particular sample based on particle count, size, type, and / or other factors to determine whether the sample is acceptable or should be discarded.

[0050] In a relatively simple embodiment, the particle detection application 814 may analyze all “slice” images of a particular container / sample (e.g., 90 images corresponding to 90 rotations of the container with the laser sheet at a fixed orientation) and label anything that appears between the container walls as a particle inside the container and anything that appears outside the container walls as a particle outside the container. However, as noted above, more sophisticated algorithms may be used. For example, the particle detection application 814 may label anything that appears between the container walls in any image as a “candidate particle” and then use a classifier (e.g., a trained neural network) to determine whether each candidate is indeed a particle or a bubble (and / or if not a bubble, to classify the type of particle, etc.). As another example, the particle detection application 814 may also analyze images from one or more additional imagers (e.g., Figure 6 Imager 604B or Figure 7 The particle detection application 814 may also or alternatively utilize any other suitable technique(s) to detect, classify, and / or locate particles within containers and samples. For example, images from both imagers may be used to better determine the three-dimensional particle position within the container at a single time corresponding to a single rotation / position of the container (this may be desirable if the sample viscosity is not very high and particles / bubbles may move slightly when the container is rotated from one position to another). The particle detection application 814 may also or alternatively utilize any other suitable technique(s) to detect, classify, and / or locate particles within containers and samples.

[0051] Fig. 9 is a flow chart of an example method 900 for imaging a container holding a sample. The method 900 may be performed by one or more portions of the imaging system 300, 500, 600, or 700 and / or by one or more portions of the automated inspection system 800. For example, block 902 may be performed by one of the laser sources 306, 506A, 606A, and 706, block 904 may be performed by one of the imagers 304, 504, 604A, and 704A, and block 906 may be performed by the computer system 802 (e.g., by the processing unit 810 when executing instructions of the particle detection application 814 stored in the memory unit 812).

[0052] At block 902, a container is illuminated with a laser sheet in a first direction corresponding to a first axis (e.g., Figure 3A The plane of the laser sheet is defined by the first axis and an orthogonal second axis (e.g., Figure 3A For example, the laser sheet may pass through the central axis of the container (e.g., Figure 3AThe laser sheet is oriented along a third axis (e.g., Figure 3A The third axis 334 of the container is associated with a certain thickness. At the location where the laser sheet enters the container, the thickness of the laser sheet can be greater than zero but less than 1 mm, less than 2 mm, less than 3 mm, less than 4 mm, less than 5 mm, etc. When stated as a range, in various embodiments, the thickness can be somewhere between 1 mm and 3 mm, somewhere between 1 mm and 5 mm, somewhere between 0.5 mm and 10 mm, etc. In some embodiments, the thickness is between 1 / 360 and 1 / 30 of the perimeter (e.g., circumference) of the container. For example, the laser sheet can include white light, or can be limited to a narrower portion of the visible spectrum (e.g., a red laser sheet).

[0053] At block 904, an image of the container when illuminated by the laser sheet is captured by an imager (e.g., imager 304, 504, 604A, or 704A). The imager has an imaging axis (e.g., Figure 3A imaging axis 324).

[0054] At block 906, the image captured at block 904 is analyzed to detect particles inside and / or on the exterior surface of the container. In some embodiments, external particles are "detected" for the sole purpose of disregarding these particles (e.g., for quality control procedures, where particles outside the container may not be of concern). Block 904 may also include classifying particles inside the container (by type, size, etc.), counting particles inside the container, and / or one or more other operations.

[0055] In some embodiments, method 900 includes Fig. 9 For example, method 900 may include a first additional block, wherein the container is moved by a plurality of rotations about a central axis of the container while illuminating the container with a laser sheet. Method 900 may also include a second additional block, wherein a plurality of images of the container are captured by an imager (each image corresponding to a respective rotation in the plurality of rotations), and may include a third additional block, wherein each of the plurality of images is analyzed to detect particles on the inner and / or outer surface of the container.

[0056] In some embodiments, the laser sheet is of a first color (e.g., red), and the imager is configured to filter out colors other than the first color. In one such embodiment, method 900 includes a first additional block, wherein the container is illuminated with a second laser sheet of a different color (e.g., blue) while the container is illuminated with the laser sheet. The second laser sheet may illuminate the container in a second direction that is not parallel to the first axis (i.e., a direction that is not parallel to the other laser sheet), and the plane of the second laser sheet may be defined by the second direction and a third direction that is substantially parallel to the second axis. Method 900 may also include a second additional block, wherein an additional image of the container is captured by an additional imager that is configured to filter out colors other than the color of the second laser sheet and has an imaging axis that is substantially orthogonal to at least the second (and possibly third) direction. For example, the additional image may be captured simultaneously with the image captured at block 904. Block 906 may then include analyzing the two images of the container to detect particles.

[0057] In yet another embodiment, the laser sheet is of a first color (e.g., red), and the imager is configured to filter out colors other than the first color, as in the above example. However, in this embodiment, method 900 includes a first additional block, wherein the container is illuminated with light of a different color (e.g., blue) while the container is illuminated with the laser sheet, the different color of light illuminating all or at least a majority of the volume / contents of the container. Method 900 may also include a second additional block, wherein an additional image of the container is captured by an additional imager, the additional imager being configured to filter out colors other than the color of the additional (e.g., non-laser) light source. For example, the additional image may be captured simultaneously with the image captured at block 904. Block 906 may then include analyzing the two images of the container to detect particles.

[0058] Although systems, methods, devices and parts thereof have been described according to exemplary embodiments, these systems, methods, devices and parts thereof are not limited thereto. The detailed description is to be interpreted as exemplary only and does not describe every possible embodiment of the present invention, because it would be impractical to describe every possible embodiment if possible. Many alternative embodiments may be implemented using current technology or technology developed after the filing date of this patent application, and these embodiments still fall within the scope of the claims defining the present invention.

[0059] Those skilled in the art will appreciate that various modifications, changes and combinations may be made to the embodiments described above without departing from the scope of the present invention, and such modifications, changes and combinations may be considered to be within the scope of the present invention.

Claims

1. A method for imaging a container containing a sample, the method comprising: illuminating the container with a laser sheet of a first color; capturing one or more images of the container with a first imager configured to filter out colors other than the first color; While illuminating the container with the laser sheet of the first color, illuminating the container with light of a second color different from the first color, wherein the light of the second color illuminates at least a majority of the entire volume of the container; capturing one or more additional images of the container with a second imager configured to filter out colors other than the second color; as well as The one or more images and the one or more additional images are analyzed by one or more processors to detect particles on the interior and / or exterior surfaces of the container.

2. The method of claim 1, wherein: Illuminating the container with the laser sheet includes illuminating the container with the laser sheet passing through a central axis of the container.

3. The method of claim 1, wherein: The laser sheet illuminates the container in a first direction corresponding to a first axis, and wherein a plane of the laser sheet is defined by the first axis and a second axis orthogonal to the first axis.

4. The method of claim 3, wherein: The first imager has an imaging axis that is substantially orthogonal to the first axis and the second axis.

5. The method of claim 3, wherein: At a location where the laser sheet enters the container and along a third axis orthogonal to the first axis and the second axis, the laser sheet has a thickness of less than 3 millimeters.

6. The method of claim 1, wherein: The thickness of the laser sheet covers between 1 / 360 and 1 / 30 of the circumference of the container.

7. The method of claim 1, wherein: The first direction is substantially orthogonal to the central axis of the container.

8. The method of claim 1, wherein: Analyzing the one or more images and the one or more additional images includes: determining that a candidate particle exists; and A classifier is used to determine whether a candidate particle is a particular type of particle.

9. The method of claim 8, wherein: The classifier is a trained machine learning model.

10. The method of claim 1, wherein: Analyzing the one or more images and the one or more additional images includes: determining that a candidate particle exists; and A classifier is used to determine whether a candidate particle is a bubble.

11. An imaging system comprising: a first illumination source configured to illuminate the container with a laser sheet of a first color; a first imager configured to (i) filter out colors other than a first color, and (ii) capture one or more images of the container; a second illumination source configured to illuminate the container with light of a second color different from the first color, wherein the light of the second color illuminates at least a majority of the overall volume of the container; a second imager configured to (i) filter out colors other than the second color, and (ii) capture one or more additional images of the container; and One or more processors are configured to analyze the one or more images and the one or more additional images to detect particles on the interior and / or exterior surfaces of the container.

12. The imaging system of claim 11, wherein: The laser sheet passes through the central axis of the container.

13. The imaging system of claim 11, wherein: The laser sheet illuminates the container in a first direction corresponding to a first axis, and wherein a plane of the laser sheet is defined by the first axis and a second axis orthogonal to the first axis.

14. The imaging system of claim 13, wherein: The first imager has an imaging axis that is substantially orthogonal to the first axis and the second axis.

15. The imaging system of claim 13, wherein: At a location where the laser sheet enters the container and along a third axis orthogonal to the first axis and the second axis, the laser sheet has a thickness of less than 3 millimeters.

16. The imaging system of claim 11, wherein: The thickness of the laser sheet covers between 1 / 360 and 1 / 30 of the circumference of the container.

17. The imaging system of claim 11, wherein: The first direction is substantially orthogonal to the central axis of the container.

18. The imaging system of claim 11, wherein: The one or more processors are configured to analyze the one or more images and the one or more additional images by at least: determining that a candidate particle exists; and A classifier is used to determine whether a candidate particle is a particular type of particle.

19. The imaging system of claim 18, wherein: The classifier is a trained machine learning model.

20. The imaging system of claim 11, wherein: The one or more processors are configured to analyze the one or more images and the one or more additional images by at least: determining that a candidate particle exists; and A classifier is used to determine whether a candidate particle is a bubble.

Images