Systems, methods and apparatus for autonomous diagnostic verification of optical components of vision-based inspection systems

Abstract

A method for autonomous diagnostic verification and detection of defects in an optical component of a vision-based inspection system is provided. The method includes: illuminating a light panel with a first light intensity pattern; capturing a first image of the first light intensity pattern with a sensor; illuminating the light panel with a second light intensity pattern different from the first light intensity pattern; capturing a second image of the second light intensity pattern with the sensor; comparing the first image and the second image to generate a comparison of the images; and identifying a defect in the light panel or the sensor based on the comparison of the images. A system adapted to perform the method is provided as a further aspect.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 733,954, filed on September 20, 2018, and entitled “METHOD AND DEVICE FOR AUTONOMOUS OPTICAL COMPONENTS HEALTH CHECK,” which is incorporated herein in its entirety for all purposes. Technical Field

[0003] The present disclosure relates to vision-based systems, and more particularly to systems, methods, and apparatus configured to perform automated pre-inspection of biological specimens. Background Art

[0004] In some vision-based automated sample inspection systems, a sample tube containing a sample (e.g., a biological sample) can be positioned at a desired location in the vision-based sample inspection system, and then one or more digital images can be obtained. Such a system can utilize optical components such as an illumination panel (e.g., a light emitting diode (LED) light panel) and an optical sensor. Based on the one or more digital images obtained, various properties of the sample and / or the sample container containing the sample can be obtained. For example, determinations of various volumes of sample components can be obtained, such as serum volume, plasma volume, sedimentation red blood cell volume of a centrifuged sample, the presence of a gel separator, the size of the sample tube, the color of the cap, and the presence of interferents, such as hemolysis (H), icterus (I), or lipemia (L) (collectively referred to as HIL). Therefore, systems, methods, and devices that exhibit robust operation are sought. Summary of the invention

[0005] In some embodiments, a method for autonomous diagnostic verification of an optical component of a vision-based inspection system is provided. The method includes: illuminating a light panel with a first light intensity pattern; capturing a first image of the first light intensity pattern with a sensor; illuminating the light panel with a second light intensity pattern different from the first light intensity pattern; capturing a second image of the second light intensity pattern with the sensor; comparing the first image and the second image to generate an image comparison; and identifying a defect in the light panel or the sensor based on the image comparison.

[0006] In some embodiments, a vision-based inspection system is provided that is operable to perform autonomous diagnostic verification of an optical component. The system includes a computer, the computer includes a memory, the memory stores instructions executable by the computer, the instructions including: illuminating a light panel with a first light intensity pattern; capturing a first image of the first light intensity pattern with a sensor; illuminating the light panel with a second light intensity pattern different from the first light intensity pattern; capturing a second image of the second light intensity pattern with the sensor; comparing the first image and the second image to generate a comparison of the images; and identifying a defect in the light panel or the sensor based on the comparison of the images.

[0007] In some embodiments, an alternative method for autonomous diagnostic verification of an optical component of a vision-based inspection system is provided. The method includes: capturing a first image of an illuminated light panel with a sensor; activating a mask on the light panel; capturing a second image of the masked light panel with the sensor; comparing the first image and the second image to generate an image comparison; and identifying a defect in the light panel or the sensor based on the image comparison.

[0008] From these and other aspects of the disclosure, numerous other aspects are provided. Other features and aspects of the disclosure will become more fully apparent from the following detailed description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1A is a schematic top view depiction of an example embodiment of a vision-based inspection system including an optical component diagnostic verification system according to an embodiment of the present disclosure.

[0010] Figure 1B is a schematic rear view of a first example LED panel assembly according to an embodiment of the present disclosure.

[0011] Figure 1C is a schematic rear view of a second example LED panel assembly according to an embodiment of the present disclosure.

[0012] Figure 2A and 2B is a schematic depiction of an LED panel assembly illuminated with two different example light intensity patterns according to an embodiment of the present disclosure.

[0013] Figure 3A is a schematic depiction of an LED panel assembly including a debris defect according to an embodiment of the present disclosure.

[0014] Figure 3B is an example of an image correlation graph generated according to an embodiment of the present disclosure.

[0015] Figure 4Ais a schematic depiction of an LED panel assembly including a scratch defect according to an embodiment of the present disclosure.

[0016] Figure 4B is an example of an image correlation graph generated according to an embodiment of the present disclosure.

[0017] Figure 5 is a flowchart depicting a first example method according to an embodiment of the present disclosure.

[0018] Figure 6 An example LCD filter according to an embodiment of the present disclosure is depicted.

[0019] Fig. 7A and 7B According to the embodiment of the present disclosure Figure 6 Schematic depiction of an LED panel assembly behind an LCD filter activated using two different example filter patterns.

[0020] Fig. 8A is a schematic depiction of an LED panel assembly including a chip defect according to an embodiment of the present disclosure.

[0021] Figure 8B is an example of an image correlation graph generated according to an embodiment of the present disclosure.

[0022] Fig. 9 is a flowchart depicting a second example method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] In a vision-based automated sample tube inspection system using optical components such as an illuminator (e.g., a light emitting diode (LED) light panel) and an optical sensor (e.g., a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) image sensor), any defects in the illuminator and / or the sensor (such as scratches or dust particles thereon) have the potential to affect the accuracy of the sample inspection. Thus, in one aspect, methods and apparatus are provided to maximize the proper operation of the optical components used. In particular, systems, methods, and apparatus are provided to enable autonomous diagnostic verification of the optical components of a vision-based inspection system.

[0024] The system and method for detecting defects in the optical components of a vision-based inspection system only uses the light source and sensor already present in the inspection system. In other words, the embodiments disclosed herein do not utilize additional hardware to test the presence of defects in the optical components of the vision-based inspection system.

[0025] The most intuitive way to inspect an optical system for defects is via direct manual visual inspection. However, this approach may not be easy to implement, especially if the optical components are mounted inside a housing, so that manual visual inspection may result in system downtime in order to disassemble the system, inspect the components, and reassemble the system. While "hot pixels" or "dead pixels" of a sensor can be easily detected by viewing captured images taken when the lighting assembly is turned on and off, it is much more difficult to detect defects on the sensor or light source (such as dust particles or scratches) by using standard images captured by the sensor.

[0026] In some embodiments where the vision-based inspection system includes an LED light panel illuminated with a plurality of independently controllable LEDs, embodiments of the present disclosure relate to using different groups of LEDs to generate a changeable light intensity pattern so as to illuminate the panel with two different (e.g., complementary) light intensity patterns, so that the captured images of the two different patterns can be compared to indicate the presence of a defect. In particular, areas that do not change between the two images can be identified as indicating the presence of a defect. In other words, if there is no defect, the image of the first light intensity pattern generated by the first group of LEDs will appear different at all points on the LED panel than the image of the second light intensity pattern generated by the second (completely different) group of LEDs. However, if there is a defect on the light panel or sensor, the area affected by the defect will not change significantly or at all compared to other areas between the images of the two different light intensity patterns. In essence, even if the background illumination changes, the defect that blocks the light will appear the same (i.e., consistent) in the two captured images, and by comparing the images of the different illumination patterns, the unchanged area can be used to identify the presence of a defect.

[0027] Figure 1A An example embodiment of a vision-based inspection system 100 is depicted. The vision-based inspection system 100 allows for obtaining multiple images of a specimen container 102 from multiple viewpoints (e.g., viewpoints 1, 2, 3), wherein each viewpoint 1, 2, and 3 includes a sensor (e.g., image capture device 104A-104C), respectively. Figure 1A , each respective viewpoint 1, 2, and 3 includes a light panel assembly 106A-106C. The vision-based inspection system 100 can be configured and adapted to automatically characterize and / or quantify a sample (e.g., including a serum or plasma portion, a sedimented blood portion, or both) and / or can additionally or alternatively quantify the sample container 102 when the sample is located at an imaging location 108. The imaging location 108 can be, for example, where the normal vectors from each of the viewpoints 1, 2, and 3 intersect.

[0028] The light panel assemblies 106A-106C of the vision-based inspection system 100 can be constructed using an array of LEDs (or other feasible light sources) optionally disposed behind or beside a diffuser and can be selectively switched. The light emitting sources of the light panel assemblies 106A-106C can emit, for example, red (R), green (G), blue (B), white light, near infrared (NIR), and / or infrared (IR) spectra. Figure 1A 104A-104C, but one or more, two or more, three or more, or even four or more such devices may be used. To minimize edge distortion, three or more sensors (e.g., image capture devices 104A-104C) may be used. Image capture devices 104A-104C may be conventional digital devices (e.g., CCD or CMOS devices) capable of capturing digital images (i.e., pixelated images). For example, three image capture devices 104A, 104B, 104C may be configured to capture images of suitable image sizes from multiple (e.g., three) different lateral viewpoints 1, 2, and 3, such as capturing all or a representative portion of a specimen container 102. As will be described herein, the captured multiple images may be processed by a computer 110, and in some instances, may be processed using "blocks" (e.g., sets of pixels, such as an 11×11 pixel grid).

[0029] Each image capture device 104A-104C can be configured and operable to obtain multiple lateral images of at least a portion of the sample container 102 and at least a portion of the sample contained therein. For example, the image capture device 104A-104C can capture a portion of the sample 212 and / or the sample container 102, including a label, a cap, and a portion of the tube. Ultimately, a 2D data set can be generated from the multiple images by each image capture device 104A-104C and stored in the memory of the computer 110. From these 2D data sets, processing of the properties of the sample and / or the sample container 102 can be performed via known methods.

[0030] In the illustrated embodiment, the plurality of image capture devices 104A-104C are shown as being arranged around the imaging position 108 and are configured to capture lateral images from a plurality of viewpoints 1, 2, and 3. When three image capture devices 104A, 104B, 104C are used, viewpoints 1, 2, and 3 may be spaced apart so that they are approximately equally spaced apart from one another, such as approximately 120 degrees from one another, as shown. As depicted, the image capture devices 104A-104C may be arranged around the edge of a track 112 that is used to move the sample container 102 into and out of the imaging position 108. Other arrangements and spacings of the plurality of image capture devices 104A-104C may be used. In this manner, an image of the sample 212 in the sample container 102 may be obtained when the sample container 102 resides in the carrier 114, and in some cases resides on the track 112, at the imaging position 108. In some embodiments, the images may overlap slightly.

[0031] The image capture devices 104A-104C can be provided in close proximity to the image window and trained or focused to capture the image window, i.e., an imaging position 108 including the expected position of the specimen container 102, the specimen container 102 can be stopped or placed in the imaging position 108 so that it is approximately centered in the viewing window. In operation, when the computer 110 receives a signal that the carrier 114 is properly located at the imaging position 108 in the vision-based inspection system 100, each image can be triggered and captured in response to a trigger signal sent by the computer 110 and provided in the communication lines 116A-116C. Each of the captured images can be processed according to methods known in the art. In particular, image processing can be used to capture and process the images so as to characterize the specimen and the specimen container 102 with a high level of detail and information content.

[0032] In some embodiments, multiple spectral images may be captured while being back-illuminated by light from light panel assemblies 106A-106C within housing 118. Spectrally switchable light sources embodied as light panel assemblies 106A-106C may back-illuminate sample container 102, such as Figure 1A As shown in , and the spectrally switchable light source includes a switchable light source arrangement as follows: the arrangement includes multiple LEDs or other light emitting devices. The capture and use of multiple color images increases the information content for analysis and can emphasize certain characteristic absorption spectra when imaging with specific spectra.

[0033] In conventional use, the light panel assemblies 106A-106C provide uniform light emission across the field of view 120A-120C of the image capture devices 104A-104C. The vision-based inspection system 100 may include a housing 118 that may at least partially surround or cover the track 112, and the sample container 102 may be located within the housing 118 during image capture operations. The housing 118 may include one or more doors 119 to allow the carrier 114 to enter the housing 118 and / or exit the housing 118.

[0034] Using the setup described above, many multiple images taken with multiple exposure times for each respective spectrum (e.g., R, G, B, white light, NIR, and / or IR) can be acquired in rapid succession, such that, for example, a whole set of images for a specimen from multiple viewpoints 1, 2, and 3 can be acquired in less than a few seconds. Other time lengths may be used.

[0035] Now turn to Figure 1B , schematically depicts a rear view of a first example embodiment of a light panel assembly 106A1. The example light panel assembly 106A1 includes a display panel 122 having two LED arrays 124, 126. In the example shown, each array 124, 126 includes two LED groups 128A, 128B, for example, arranged in an alternating pattern. In some embodiments, the two groups 128A, 128B are the same LEDs, but are separately controllable. In other words, for example, group 128A can be turned on while group 128B is turned off, and group 128A can be turned off while group 128B is turned on. Figure 1C A second example light panel assembly 106A2 is depicted, which includes a diffuser panel 122 having four LED arrays 130, 132, 134, 138. In the example shown, each array 130, 132, 134, 138 includes two LED groups 138A, 138B arranged in an alternating pattern. In some embodiments, the two groups 138A, 138B are identical LEDs, but are separately controllable. In other words, for example, group 138A can be turned on while group 138B is turned off, and group 138A can be turned off while group 138B is turned on. Other arrangements of LEDs or light sources may be used.

[0036] In some embodiments, different groups 128A, 128B (or 138A, 138B) can be controlled to emit different wavelengths of light and / or different intensities of light. Thus, for example, light panel assembly 106A2 can be used to generate a light beam by turning on LED group 138A and turning off LED group 138B. Figure 2AThe same light panel assembly 106A2 can also be used to generate a second light intensity pattern 200' different from the first light intensity pattern 200 by turning off the LED group 138A and turning on the LED group 138B, such as Figure 2B As shown on the light panel assembly 106A2. Therefore, by switching different LED groups, the intensity distribution on the panel can be changed accordingly. In some embodiments, different intensities (i.e., different current levels sent to LED groups 138A, 138B) can be used to generate the two different patterns shown. It is noted that, for example, the patterns shown are complementary. In other words, in areas where one pattern is dark, the other pattern is bright, and vice versa. Although light intensity patterns other than fully complementary can be used, in some embodiments, complementary patterns can more easily reveal defects.

[0037] In diagnostic mode and in the absence of a sample container 102 at the image location 108, the system 100 can be configured to focus the image capture device 104A on the light panel assembly 106A. When debris (e.g., particles, etc.) is present on the light panel assembly 106A or scratches or other damage is present on the light panel assembly 106A or the sensor (e.g., the image capture device 104A), the light intensity distribution around the defective area will resist changes in the LED pattern. Based on this recognition, embodiments of the present disclosure use consistency checks between images of the light panel assembly 106A illuminated with patterns of different light intensities (and / or wavelengths) to detect defects in the light panel assembly 106A or the sensor.

[0038] For example, the computer 110 may use the image capture device 104A to capture two images of the light panel assembly 106A illuminated with two patterns 200, 200' of different light intensities. The computer 110 then processes the images by dividing each image into a grid having a plurality of relatively small blocks, and for each corresponding block, calculates the cross-correlation between the images. That is, for the two images " I "and" J "Each block in" B ", using blocks B Each pixel within p The correlation is calculated by using the normalized cross-correlation of , as follows:

[0039] .

[0040] The higher the correlation calculated according to the above formula, the higher the block B” The higher the probability of containing a defect in the optical component. In operation, a threshold correlation value can be set, wherein any calculated correlation above the threshold correlation value is considered to indicate the presence of a defect in the optical component at that location.

[0041] Figure 3A An example of a light panel assembly 300 is depicted having a debris particle 302 as a defect on the panel 300 , wherein the problematic area is darker (ie, less light intensity level) than other areas of the panel 300 . Figure 4A An example of a light panel assembly 400 is depicted having one or more scratches 402 as defects on a display panel of the panel 400, wherein the problematic area is brighter (ie, greater light intensity level) than other areas.

[0042] Using the method of the present disclosure, two types of defects can be easily detected, such as Figure 3B As shown in the generated image correlation map 304 (ie, dark area 306), and as shown in Figure 4B 406). It is noted that the generated image correlation graphs 304, 404 represent a plot of calculated correlations, wherein correlation values ​​below a predetermined threshold correlation value result in the corresponding blocks being displayed as relatively light colors, and wherein correlation values ​​above the predetermined threshold correlation value result in the corresponding blocks being displayed as relatively dark colors.

[0043] Figure 5 1 is a flow chart depicting an example method 500 of using the system 100 as described above to test an optical component for the presence of defects therein or thereon. At 502, a first light intensity pattern (such as, for example, Figure 2A 1 ) to illuminate a light panel (such as, for example, light panel assembly 104A) and capture a first image of first light intensity pattern 200 using a sensor (such as, for example, image capture device 106A of FIG. 1 ). Then, at 504, a second light intensity pattern (such as, for example, Figure 2B The method further comprises: illuminating the light panel with a light intensity pattern 200' of the first image and capturing a second image of the second light intensity pattern 200' using the sensor. Next, at 506, the two captured images (the first image and the second image) are compared (e.g., by calculating a correlation between each corresponding block of the two images) to generate a comparison (e.g., a mathematical comparison) of the images (e.g., an image correlation map as described above). Based on the comparison of the images (i.e., the presence of a relatively high correlation area exceeding a predetermined threshold), the method identifies the presence of a defect in the light panel, the sensor, at 508.

[0044] In some embodiments, the light intensity pattern can be changed by inserting filters or by using a changeable filter (e.g., a configurable light blocking LCD mask) placed in front of the light panel. This can be an alternative or supplement to changing the intensity or wavelength of the LEDs in the light panel assembly.

[0045] Figure 6 A schematic representation of an example of a configurable LCD mask 600 adapted to be mounted in front of a light panel assembly 104A is depicted. The LCD mask 400 includes a plurality of LCD segments 602 that can be selectively activated (e.g., individually controlled segments 602 or groups of segments 602 that are controlled together) to block light emitted by the light panel assembly 104A or to pass light emitted by the light panel assembly 104A.

[0046] Fig. 7A and 7B Depicted are two different light intensity patterns 700, 702 generated by selectively activating two different groups of LCD segments 602 of the LCD mask 600. In an alternative embodiment, the two different light intensity patterns may be generated by simply activating all LCD segments 602 for one image and activating no LCD segments for a second image. In some embodiments, LCD segments may be used in conjunction with varying groups of LEDs.

[0047] Fig. 8A A light panel assembly 800 is depicted that includes a debris particle 302 as a defect on the light panel assembly 800 . Figure 8B A plot showing a correlation image between an image of the light panel assembly 800 captured with the LCD segments 602 activated and an image captured with the LCD segments 602 not activated. Note that where debris particles 302 are present in the image, there is a corresponding high correlation above a predetermined correlation threshold, which results in a dark region 306, and there is a low correlation below the predetermined correlation threshold, which results in the remaining region being displayed in a bright color.

[0048] Fig. 96 is a flow chart depicting a second method 900 for detecting defects in an optical component of a vision-based inspection system 100 using a filter (e.g., LCD mask 600). At 902, a first image of an illuminated LED panel behind a first LCD filter pattern is captured. At 904, a second image of the illuminated LED panel behind a second LCD filter pattern is captured. Note that in some embodiments, the second LCD filter pattern may simply be the absence of any LCD filter pattern at all. Next, at 906, the two captured images are compared (e.g., by calculating a correlation between each corresponding block of the images) to generate a comparison (e.g., a mathematical comparison) of the images (e.g., an image correlation map as described above). At 908, based on the comparison of the images (e.g., the presence of a relatively high correlation region exceeding a predetermined threshold), the presence of a defect in the light panel or sensor is identified.

[0049] An advantage of the embodiments disclosed herein is that various types of defects of optical components can be easily detected. Since no tools are used other than the components of the vision-based inspection system, the diagnostic methods disclosed herein can be performed autonomously without shutting down the system and / or removing the housing. Thus, one or more defect inspections can be performed before and / or after taking an image or a series of screening images of a specimen and / or specimen container to confirm acceptable operation of the optical component.

[0050] Although embodiments are described herein with reference to specific examples, the scope of the disclosure is not intended to be limited to the details described. Rather, various modifications may be made in the details within the scope and range of equivalents of the disclosure.

Claims

1. A method for autonomous diagnostic verification of an optical assembly of a vision-based inspection system including a light panel and a sensor, the method comprising: illuminating the light panel with a first light intensity pattern; capturing, with a sensor, a first image of a first light intensity pattern; illuminating the light panel with a second light intensity pattern different from the first light intensity pattern; capturing, with the sensor, a second image of the second light intensity pattern; comparing the first image and the second image to generate a comparison of the first image and the second image; as well as identifying a defect in the light panel or the sensor based on a comparison of the first image and the second image, wherein the light panel comprises an LED light panel illuminated by a plurality of independently controllable LEDs, and wherein the first light intensity pattern and the second light intensity pattern are generated using two different sets of LEDs, and Wherein comparing the first image and the second image comprises: calculating correlations between portions of the first image and the second image, and wherein portions having correlations between the first image and the second image above a predetermined threshold are identified as indicating the presence of the defect. 2 . The method of claim 1 , wherein the portions are defined regions of the first image and the second image, the regions comprising blocks. 3 . The method of claim 2 , wherein the blocks are arranged in a grid overlaying the first image and the second image.

4. The method according to claim 3, wherein the correlation is calculated based on the following equation: Wherein B represents the tested area, I represents the first image, J represents the second image, and p represents each pixel belonging to block B.

5. The method of claim 4, wherein the defect is identified if the calculated correlation for a given block exceeds a predetermined threshold. The method of claim 1 , wherein the first light intensity pattern is complementary to the second light intensity pattern.

7. A system for autonomous diagnostic verification of an optical assembly of a vision-based inspection system including a light panel and a sensor, the system comprising: A computer comprising a memory storing instructions executable by the computer to implement the following steps: illuminating the light panel with a first light intensity pattern; capturing, with a sensor, a first image of a first light intensity pattern; illuminating the light panel with a second light intensity pattern different from the first light intensity pattern; capturing, with the sensor, a second image of the second light intensity pattern; comparing the first image and the second image to generate a comparison of the first image and the second image; as well as identifying a defect in the light panel or the sensor based on a comparison of the first image and the second image, wherein the light panel comprises an LED light panel illuminated by a plurality of independently controllable LEDs, and wherein the first light intensity pattern and the second light intensity pattern are generated using two different sets of LEDs, and Wherein comparing the first image and the second image comprises: calculating correlations between portions of the first image and the second image, and wherein portions having correlations between the first image and the second image above a predetermined threshold are identified as indicating the presence of the defect.

8. The system of claim 7, wherein the portions are defined regions of the first image and the second image, the regions comprising blocks.

9. The system of claim 8, wherein the tiles are arranged in a grid that overlays the first image and the second image.

10. The system of claim 9, wherein the correlation is calculated based on the following equation: Wherein B represents the tested area, I represents the first image, J represents the second image, and p represents each pixel belonging to block B.

11. The system of claim 10, wherein the defect is identified if the calculated correlation for a given block exceeds a predetermined threshold.

12. The system of claim 7, wherein the first light intensity pattern is complementary to the second light intensity pattern.

13. A method for autonomous diagnostic verification of an optical assembly of a vision-based inspection system including a light panel and a sensor, the method comprising: activating a first set of segments of a mask on a light panel; capturing, with a sensor, a first image of the light panel illuminated when the first set of segments are activated; activating a second set of segments of the mask on the light panel that is different from the first set of segments; capturing, with the sensor, a second image of the light panel illuminated when the second set of segments is activated; comparing the first image and the second image to generate a comparison of the first image and the second image; as well as identifying a defect in the light panel or the sensor based on a comparison of the first image and the second image, Wherein comparing the first image and the second image comprises: calculating correlations between portions of the first image and the second image, and wherein portions having correlations between the first image and the second image above a predetermined threshold are identified as indicating the presence of the defect.

14. The method of claim 13, wherein the portions are defined regions of the first image and the second image, the regions comprising blocks.

15. The method of claim 14, wherein the tiles are arranged in a grid overlaying the first image and the second image.

16. The method of claim 15, wherein the correlation is calculated based on the following equation: Where B represents the tested area, I represents the first image, J represents the second image, and p represents each pixel belonging to block B.

17. The method of claim 16, wherein the defect is identified if the calculated correlation for a given block exceeds a predetermined threshold.

Images