Assembly line incorporated with electronic vision inspection

Abstract

To provide a new technique for product assembly capable of reducing errors and improving efficiency of package assembly.SOLUTION: A method includes: obtaining an image of an object including two regions, determining a number and an orientation of one or more patterns in each of the two regions; determining a number of one or more items having the one or more patterns based on the determined number and orientation of the one or more patterns; comparing the determined number of one or the more items to an expected number of the one or more items in each of the two regions; and generating a pass signal or a fail signal based on the comparison, and moving a belt forward including an object having the one or more patterns in the two regions based on receiving the pass signal.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an assembly line incorporating electronic visual inspection.

Background Art

[0002] The assembly of products containing multiple articles, such as pharmaceutical packaging, is a complex task. Assembly can be carried out in one or more stages using the articles to be placed in the product at each stage. Errors can occur at each stage by misplacing the correct article in the product, overloading the product with too many of the correct article, and / or inserting the wrong article into the product. Products shipped in a state where an error has finally occurred result in loss of revenue, increased customer complaints, and loss of time in dealing with customer complaints. In the case of pharmaceutical packages, one of the unintended consequences of improper packaging is that clinicians or patients may not want to use the pharmaceuticals contained in the improperly assembled package. This can be particularly true for pharmaceuticals administered parenterally, such as subcutaneously, intramuscularly, intravenously, intravitreally, or by inhalation. Even if an improperly assembled package is returned by a clinician or patient to the manufacturer, regulatory agencies such as the US Food and Drug Administration do not permit repackaging of pharmaceuticals, resulting in a Notice of Event (NOE). Such an NOE triggers an investigation, increases costs, and potentially results in a loss of competitiveness.

Summary of the Invention

Problems to be Solved by the Invention

[0003] Therefore, it would be desirable to develop new technologies for product assembly that overcome these and other limitations of the prior art, reduce errors, and enhance it by increasing the efficiency of package assembly.

Means for Solving the Problems

[0004] The following general description and the following forms for carrying out the invention are both exemplary and explanatory only, and should not be construed as limiting. A method and system are disclosed for obtaining a first image of a tray, determining the presence or absence of one or more first patterns in the first image, determining the rotation of each of the one or more first patterns in the first image, and performing an operation based on the presence or absence and rotation of the one or more first patterns in the first image.

[0005] Further advantages are partly described in the following description or can be learned by practice. These advantages will be realized and achieved by the elements and combinations particularly pointed out in the appended claims.

Brief Description of the Drawings

[0006] The accompanying drawings incorporated herein and constituting a part of this specification illustrate embodiments and, together with the description, serve to explain the principles of the method and system.

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 4A

Figure 4B

Figure 5A

Figure 5B

Figure 6A

Figure 6B

Figure 7A

Figure 7B

Figure 8A

Figure 8B

Figure 9

Figure 10

Figure 11

Best Mode for Carrying Out the Invention

[0007] Before the method and system are disclosed and described, it should be understood that the method and system are not limited to a particular method, particular components, or particular embodiments. It should be further understood that the terms used herein are for the purpose of describing particular embodiments and are not intended to be limiting.

[0008] Furthermore, as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Ranges may be expressed herein as from "about" a particular value and / or to "about" another particular value. When such a range is expressed, other embodiments include from the one particular value and / or to the other particular value. Similarly, when values are expressed using the preceding term "about," the particular value forms another embodiment. It will also be further understood that each endpoint of each range is meaningful in relation to and independent of the other endpoint.

[0009] "Any" or "optionally" means that the event or circumstance recited may or may not occur, and that the description includes instances where the event or circumstance occurs and instances where it does not.

[0010] Throughout the description and claims of this specification, the terms "comprise", "comprising", "comprises" and variations thereof mean "include but are not limited to", and for example, are not intended to exclude other components, integers or steps. "Exemplary" means "an example of", and is not intended to convey a suggestion of a preferred or ideal embodiment. "Such as" is used for illustrative purposes, not in a limiting sense.

[0011] Components that can be used to implement the disclosed methods and systems are disclosed. These and other components are disclosed herein, and when combinations, subsets, interactions, groups, etc. of these components are disclosed, specific citations of each of the various individual, collective, combinations and permutations of these need not be explicitly disclosed, but each is understood to be specifically contemplated and described herein for all methods and systems. This applies to all aspects of this application, including but not limited to the steps in the disclosed methods. Thus, it should be understood that if there are various additional steps that can be performed, each of these additional steps can be performed using any particular embodiment or combination of embodiments of the disclosed method.

[0012] The method and system may be more readily understood by reference to the following detailed description of the preferred embodiments, the examples included therein, the drawings, and the description before and after them.

[0013] As will be understood by those skilled in the art, the present method and system can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Further, the present method and system can take the form of a computer program product in a computer-readable storage medium having computer-readable program instructions (e.g., computer software) embodied therein. More specifically, the present method and system can take the form of computer software implemented on the web. Any suitable computer-readable storage medium can be utilized, including a hard disk, CD-ROM, optical storage device, or magnetic storage device.

[0014] Embodiments of the present method and system are described below with reference to block diagrams and flowchart diagrams of methods, systems, apparatuses, and computer program products. It will be understood that each block of the block diagrams and flowchart diagrams, as well as combinations of blocks in the block diagrams and flowchart diagrams, can be implemented by computer program instructions. These computer program instructions are loaded onto a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to create means for causing the instructions executed on the computer or other programmable data processing apparatus to implement the functions specified in one or more blocks of the flowchart, thereby manufacturing a machine.

[0015] These computer program instructions can also be stored in a computer-readable memory that directs a computer or other programmable data processing apparatus to function in a particular manner so as to produce an article of manufacture including computer-readable instructions for performing the functions specified in one or more blocks of the flowchart. The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus, and to generate a process that is executed by the computer to provide steps for performing the functions specified in one or more blocks of the flowchart.

[0016] Accordingly, the blocks of the block diagrams and flowchart diagrams support combinations of means for performing the specified functions, combinations of steps for performing the specified functions, and program instruction means for performing the specified functions. It will also be understood that each block of the block diagrams and flowchart diagrams, and combinations of blocks in the block diagrams and flowchart diagrams, can be implemented by dedicated hardware-based computer systems that perform the specified functions or steps, or by combinations of dedicated hardware and computer instructions.

[0017] The present disclosure relates to improvements in computer functionality related to manufacturing and product assembly. FIG. 1 is a block diagram showing various aspects of an exemplary system 100 in which the method and system can operate. Those skilled in the art will understand that functional descriptions are provided herein and that each function can be executed by software, hardware, or a combination of software and hardware.

[0018] In one aspect, system 100 can include a conveyor belt 101. The conveyor belt 101 can include one or more anti-slip elements 102. The one or more anti-slip elements 102 can be made of rubber or a similar material for attachment to the conveyor belt 101. The one or more anti-slip elements 102 are raised or otherwise extend on the surface of the conveyor belt 101. The one or more anti-slip elements 102 can include a front anti-slip element and a rear anti-slip element based on the moving direction 103. The front anti-slip element and the rear anti-slip element can be relative to an object placed on the belt such that the front anti-slip element is in front of the object with respect to the moving direction 103 and the rear anti-slip element is behind the object with respect to the moving direction 103. Thus, the front anti-slip element for a first object can be the rear anti-slip element for a second object in front of the first object, etc. One or more objects 104 can be placed on the conveyor belt 101. In one aspect, the one or more objects 104 can include one or more assembled products. For example, the one or more objects 104 can include a tray. The tray can be configured to hold one or more articles. The one or more articles can be related to treatment. For example, the one or more articles can include one or more syringes, auto-injectors, one or more needles for syringes, one or more containers for pharmaceuticals, one or more sets of pamphlets or instructions, combinations thereof, etc.

[0019] In one aspect, the set of instructions describes information regarding the method of use and administration of the drug. In another aspect, the instructions are a drug label approved by a regulatory agency such as the US Food and Drug Administration.

[0020] In one aspect, the drug is a solid formulation. In another aspect, the drug is a liquid formulation. In another aspect, the drug is a gel formulation. In one aspect, the agent is formulated for oral administration. In another aspect, the agent is formulated for parenteral administration. In another aspect, the agent is formulated for subcutaneous administration. In another aspect, the agent is formulated for intramuscular administration. In another aspect, the agent is formulated for intravenous administration. In another aspect, the agent is formulated for inhalation administration. In another aspect, the agent is formulated for intraocular administration.

[0021] In one aspect, the agent comprises a small molecule active ingredient. In another aspect, the agent comprises a biological preparation. In another aspect, the agent comprises a peptide or polypeptide active ingredient. In one aspect, the agent comprises a vascular endothelial growth factor (VEGF) derivative active ingredient. In another aspect, the agent comprises aflibercept as described in one or more of U.S. Patent Nos. 7,070,959; 7,303,746; 7,303,747; 7,306,799; 7,374,757; 7,374,758; 7,531,173; 7,608,261; 7,972,598; 8,029,791; 8,092,803; 8,343,737; 8,647,842, each of which is incorporated by reference in its entirety.

[0022] The conveyor belt 101 can pass over a drive roll that can be driven by a stepping motor 105. The use of the stepping motor 105 enables the accurate positioning of one or more objects 104 with respect to cameras 106, 107, and 108. The respective lengths of the one or more objects 104 can be represented as an exact number of motor steps. The conveyor belt 101 can accurately move forward or backward to move each of the one or more objects 104 into the fields of view 109, 110, and 111 associated with cameras 106, 107, and 108. A programmable logic controller (PLC) 112 (the PLC 112 can comprise a computing device, a PLC, or other controller / processor) can be configured to cause the stepping motor 105 to execute any number of steps in either direction to move the one or more objects 104 into the fields of view 109, 110, and 111.

[0023] In one aspect, camera 106, camera 107, and / or camera 108 can be configured to scan, decode, read, detect, image, capture, and / or interpret visual codes. In some aspects, camera 106, camera 107, and / or camera 108 can be configured to process laser, linear, or area imaging. For example, in one aspect, examples of camera 106, camera 107, and / or camera 108 can include an imager for scanning, reading, and decoding one-dimensional or two-dimensional barcodes. Examples of camera 106, camera 107, and / or camera 108 can include any imager, barcode scanner, or visual code scanner that can extract information from visual codes consistent with the disclosed embodiments. In certain aspects, camera 106, camera 107, and / or camera 108 can be configured to process scanned barcodes, images, and other data. Examples of camera 106, camera 107, and / or camera 108 can include one or more depth cameras for capturing, processing, detecting, observing, modeling, detecting, and interacting with a three-dimensional environment. In certain aspects, camera 106, camera 107, and / or camera 108 can recognize and detect the depth and color of objects within fields of view 109, 110, and 111, respectively. Camera 106, camera 107, and / or camera 108 can also provide other camera and video recorder functions such as taking photos, recording videos, streaming images or other data, storing data in an image buffer, etc. These functions may or may not include depth information. In relation to the disclosed hardware and / or software processes, camera 106, camera 107, and / or camera 108 can determine the size, orientation, and visual characteristics of one or more objects 104. Camera 106, camera 107, and / or camera 108 can include or embody any camera known to those skilled in the art that can process the processes disclosed herein.Camera 106, Camera 107 and / or Camera 108 can include appropriate hardware and software components (e.g., circuits, software instructions, etc.) for transmitting signals and information to and from the Pass / Fail Controller 113 to perform a process consistent with the disclosed embodiments. The Pass / Fail Controller 113 can include a computing device, a PLC, or other controller / processor. Camera 106, Camera 107 and / or Camera 108 can each output an image and / or one or more notifications to Monitor 114, Monitor 115 and Monitor 116 respectively.

[0024] Positioning within the fields of view 109, 110 and 111 of one or more objects 104 can be done at startup of the system 100 and can also be adjusted during use of the system 100. One or more of Camera 106, Camera 107 and / or Camera 108 can be used to ensure proper positioning of the conveyor belt 101. For example, Camera 107 can be configured to generate an image of an area within the field of view 110. Camera 107 can determine the position of one or more anti-slip members 102 in the image. In one aspect, Camera 107 can determine the position of the front anti-slip member. Camera 107 can compare the determined position of one or more anti-slip members 102 in the image with a reference position. If the determined position is equal to the reference position, there is no need to adjust the conveyor belt 101. If the determined position is not equal to the reference position, Camera 107 can determine an offset based on the difference between the determined position and the reference position. The offset can be determined in terms of a measure of distance, e.g., millimeters, centimeters, inches, etc., and / or the offset can be determined in terms of a number of steps. Camera 107 can send a signal to the PLC 112 to move the conveyor belt 101 forward or backward by the amount of the offset by actuating the stepper motor 105.

[0025] During operation, system 100 can be configured to evaluate the current assembled state of one or more objects 104 and take one or more actions based on the current assembled state. As each of the one or more objects 104 advances by conveyor belt 101, each of the one or more objects 104 is placed within the respective fields of view 109, 110, and 111 of camera 106, camera 107, and / or camera 108. Although FIG. 1 only shows three cameras, it is particularly contemplated that fewer than three or more than four cameras can be used. It is further contemplated that conveyor belt 101 can be configured to have four or more of the illustrated objects 104 placed thereon, regardless of the number of cameras. As the one or more objects 104 travel along conveyor belt 101, one or more articles can be assembled onto the one or more objects 104 by a human operator or a robot.

[0026] When one or more of the objects 104 are within the field of view of one of the cameras, the camera can generate an image of the object 104 within the field of view associated with that camera. For example, camera 106 can generate an image of the area within field of view 109, camera 107 can generate an image of the area within field of view 110, and camera 108 can generate an image of the area within field of view 111. Each of cameras 106, 107, and / or 108 can analyze their respective images. Analyzing the images can include determining the presence or absence of one or more patterns. The one or more patterns can include text patterns, numeric patterns, symbol patterns, and combinations thereof. For example, a text pattern can include any string such as "filter needle". A numeric pattern can include any sequence of numbers such as "6941518". A symbol pattern can include any sequence of symbols such as "●□□◆". In one aspect, cameras 106, 107, and / or 108 can utilize optical character recognition (OCR) to "read" one or more patterns. In another aspect, cameras 106, 107, and / or 108 can be configured not to utilize OCR, but rather to simply recognize one or more patterns as a specific pattern.

[0027] In one aspect, one or more patterns can be embodied on one or more articles that are assembled into one or more objects 104. In one aspect, at least a portion of the one or more articles can include one or more associated patterns. Thus, when cameras 106, 107, and / or camera 108 determine the presence of one or more patterns, the presence of the one or more patterns indicates the presence of an article associated with a particular pattern. For example, if camera 106 determines the presence of a "filter needle" in an image of an area within the field of view 109, it can be concluded that an article associated with the pattern "filter needle" is present on the object 104 within the field of view 109. Cameras 106, 107, and / or camera 108 can be configured to determine the presence or absence of multiple patterns in a single image. For example, camera 106 can determine the presence of a "filter needle" and a "filter needle" in an image of an area within the field of view 109. The presence of both patterns can indicate that an article associated with the pattern "filter needle" that appears twice is present on the object 104 within the field of view 109.

[0028] Articles that can be assembled into one or more objects 104 can each be associated with one or more patterns indicating the presence or absence of a specific number of articles. For example, an article may be embodied with a specific pattern appearing only once. If cameras 106, 107, and / or 108 determine that a specific pattern appears only once, it can be concluded that only one article is present. However, if cameras 106, 107, and / or 108 determine that a specific pattern appears two or more times, it can be concluded that multiple articles are present. In another example, an article may be embodied with a specific pattern appearing twice. If cameras 106, 107, and / or 108 determine that a specific pattern appears only twice, it can be concluded that only one article is present. However, if cameras 106, 107, and / or 108 determine that a specific pattern appears once or three or more times, it can be concluded that multiple articles are present. In a further example, an article may be embodied with a specific pattern within a certain range. For example, an article may be embodied with a specific pattern appearing 1 to 2 times. If cameras 106, 107, and / or 108 determine that a specific pattern occurs once or twice, it can be concluded that only one article is present. However, if cameras 106, 107, and / or 108 determine that a specific pattern appears three or more times, it can be concluded that multiple articles are present.

[0029] Camera 106, camera 107, and / or camera 108 can each be configured to analyze the entire image or one or more specific regions of the image. FIG. 2 shows an exemplary image 200 of object 104. Object 104 can include a tray 201 configured to store one or more articles. One or more articles can be assembled into tray 201 such that at least a portion of the one or more articles is present in one or more specific regions. Tray 201 can include one or more regions, such as region 202, region 203, and region 204. Region 202, region 203, and region 204 can each be associated with a region where one or more patterns should be present when an article is present within tray 201. For example, region 202 can be associated with the position of the vial cap of a vial when assembled into tray 201, region 203 can be associated with the position of one or more syringes and / or one or more needles when assembled into tray 201, and region 204 can be associated with the position of one or more pamphlets when assembled into tray 201. Camera 106, camera 107, and / or camera 108 can each be configured to analyze one or more assigned regions of image 200. For example, camera 106 can be assigned to analyze region 202 and region 203, camera 107 can be assigned to analyze region 203, and camera 108 can be assigned to analyze region 203 and region 204. Any combination of assigned regions is possible. Further, camera 106, camera 107, and / or camera 108 can each be configured to determine the presence or absence of one or more assigned patterns in the assigned regions.For example, camera 106 may be assigned to determine the presence or absence of the vial cap within region 202 and the presence or absence of the first pattern within region 203 (including the number of occurrences of the first pattern), camera 107 may be assigned to determine the presence or absence of the second pattern within region 203 (including the number of occurrences of the second pattern), and camera 108 may be assigned to determine the presence or absence of the third pattern within region 203 (including the number of occurrences of the third pattern) and the presence or absence of the fourth pattern within region 204 (including the number of occurrences of the fourth pattern). Any combination of the assigned patterns and assigned regions is conceivable.

[0030] Returning to FIG. 1, each of one or more objects 104 can be configured to contain a specific number of each of one or more articles. The presence of each article in the specific number indicates that the one or more objects 104 are correctly assembled. The presence of something other than each article in the specific number indicates that the one or more objects 104 are incorrectly assembled. Each of camera 106, camera 107, and / or camera 108 can be configured to perform an independent evaluation of the object 104 within the corresponding field of view. If the camera determines that there is a specific number of articles configured to be detected by the camera, the camera can send a PASS signal to the pass / fail controller 113. If the camera determines that there is no specific number of articles configured to be detected by the camera, the camera can send a FAIL signal to the pass / fail controller 113. When each of camera 106, camera 107, and / or camera 108 sends a PASS signal to the pass / fail controller 113, the pass / fail controller 113 can provide a signal to the PLC 112 to advance the conveyor belt 101 by the stepping motor 105 to advance the one or more objects 104 and place them under the next camera's field of view. The pass / fail controller 113 can further send a notification to each of the monitors 114-116 to display a PASS notification. When one or more of camera 106, camera 107, and / or camera 108 sends a FAIL signal to the pass / fail controller 113, the pass / fail controller 113 does not provide a signal to the PLC 112 to advance the stepping motor 105. The pass / fail controller 113 can further send a notification to the monitors 114-116 associated with the camera that sends the FAIL signal to display a FAIL notification. An operator (e.g., a human or a robot) placed at the monitors 114-116 that display the FAIL notification can perform a corrective action to improve the FAIL state.For example, if a FAIL signal is issued as a result of a shortage, the operator can replace the shortage, and as a result, the camera that made the previous FAIL determination can regenerate and re-analyze the image, determine that the article now exists, and issue a PASS signal to the pass / fail controller 113. In another embodiment, if the FAIL signal is issued as a result of one or more extra articles, the operator removes the one or more extra articles, and as a result, the camera that made the previous FAIL determination can regenerate and re-analyze the image, determine that the required number of articles now exists, and issue a PASS signal to the pass / fail controller 113.

[0031] In a further aspect, the analysis of the images by camera 106, camera 107, and / or camera 108 can include not only determining the presence or absence of one or more patterns, but also determining the rotation of multiple patterns. In one aspect, the multiple patterns can be embodied on one or more articles that are assembled onto one or more objects 104 along a particular axis. In one aspect, at least a portion of the one or more articles can include multiple associated patterns along a particular axis. Thus, when cameras 106, 107, and / or 108 determine the presence of multiple patterns along a particular axis, the presence of the multiple patterns along the particular axis indicates the presence of an article associated with a particular pattern along the particular axis. For example, if camera 106 determines the presence of "filter needles" and "filter needles" along the same axis (e.g., 30°, 60°, 90°, 120°, 180°, etc.) in an image of an area within the field of view 109, it can be concluded that an article associated with the pattern "filter needles" and "filter needles" along the same axis is present on the object 104 within the field of view 109. Cameras 106, 107, and / or 108 can be configured to determine the rotation of multiple patterns in a single image. For example, camera 106 can determine the presence of "filter needles" and "filter needles" along a first axis, and the presence of "syringe needles" and "syringe needles" along a second axis, in an image of an area within the field of view 109. The presence of both patterns along two different axes can indicate that an article associated with two occurrences of the pattern "filter needles" along the first axis is present on the object 104, and that an article associated with two occurrences of the pattern "syringe needles" along the second axis is also present on the object 104. As a further example, camera 106 can determine the presence of "filter needles" and "filter needles" along a first axis, and the presence of "filter needles" along a second axis, in an image of an area within the field of view 109. The presence of both patterns along two different axes can indicate that two occurrences of an article associated with the pattern "filter needles" are present on the object 104.

[0032] Articles that can be assembled into one or more objects 104 can each be associated with one or more patterns embodied along a particular axis that indicate the presence or absence of a particular number of articles. For example, an article may be embodied with a particular pattern appearing twice along a particular axis. If cameras 106, 107, and / or 108 determine that a particular pattern appears only twice along a particular axis, a conclusion can be drawn that only one article is present. However, if cameras 106, 107, and / or 108 determine that a particular pattern appears along multiple axes, a conclusion can be drawn that multiple articles are present.

[0033] Figures 3A and 3B show exemplary images 300 and 303 of tray 201 including article 301 and article 302. For example, article 301 may be a vial and article 302 may be a filter needle. Any of cameras 106, 107, and / or 108 that generate image 300 can determine that a vial cap is present within region 202. The presence of a single vial cap indicates the presence of article 301. Cameras 106, 107, and / or 108 that generate image 300 can determine that there are two occurrences of the pattern "Text A" within region 203. In one aspect, the two occurrences of the pattern "Text A" can indicate the presence of one or more instances of article 302, and cameras 106, 107, and / or 108 can generate a PASS signal or a FAIL signal as needed. In another aspect, depending on the pattern configuration on article 302 (e.g., a single instance of article 302 can have either one occurrence of "Text A" or a double occurrence of "Text A"), cameras 106, 107, and / or 108 can determine whether "Text A" and "Text A" appear on the same axis. If "Text A" and "Text A" appear on the same axis, cameras 106, 107, and / or 108 can determine that a single instance of article 302 is present, and cameras 106, 107, and / or 108 can generate a PASS signal or a FAIL signal as needed. If "Text A" and "Text A" appear on different axes, cameras 106, 107, and / or 108 can determine that multiple instances of article 302 are present, and cameras 106, 107, and / or 108 can generate a PASS signal or a FAIL signal as needed. In one aspect, the determination of the axis is used to confirm the presence of any number of articles 302 and generate a PASS signal or a FAIL signal based on the predicted number of instances of article 302 versus the determined number of instances of article 302.

[0034] FIG. 4A shows an exemplary image 400 of a tray 201 that includes two instances of articles 301 and 302. The cameras 106, 107, and / or 108 that generate the image 400 can determine that there are three occurrences of a pattern (“Text A”) within the region 203. In one aspect, the three occurrences of the pattern “Text A” can indicate that one or more instances of article 302 are present, and the cameras 106, 107, and / or 108 can generate a PASS signal or a FAIL signal as needed. In another aspect, depending on the pattern configuration on article 302 (e.g., a single instance of article 302 can have one occurrence of “Text A,” a double occurrence of “Text A,” or a triple occurrence of “Text A”), the cameras 106, 107, and / or 108 can determine whether the three occurrences of “Text A” appear on the same axis. As shown in FIG. 4A, two occurrences of “Text A” appear on the same axis and one occurrence of “Text A” appears on a different axis. Thus, the cameras 106, 107, and / or 108 can determine that multiple instances of article 302 are present, and the cameras 106, 107, and / or 108 can generate a PASS signal or a FAIL signal as needed. In one aspect, the determination of the axis is used to confirm that any number of articles 302 are present and a PASS signal or a FAIL signal can be generated based on the predicted number of instances of article 302 versus the determined number of instances of article 302.

[0035] Figure 4B shows an exemplary image 401 of tray 201 that includes one instance of article 301, one instance of article 302, and one instance of article 402. In one aspect, cameras 106, 107, and / or 108 that generate image 400 can determine that there are two occurrences of a first pattern (“Text A”) within region 203 and one occurrence of a second pattern (“Text B”). In one aspect, the two occurrences of pattern “Text A” can indicate that one or more instances of article 302 are present, and cameras 106, 107, and / or 108 can generate a PASS signal or a FAIL signal as needed. In another aspect, depending on the pattern configuration on article 302 (e.g., a single instance of article 302 can have one occurrence of “Text A”, a double occurrence of “Text A”, or a triple occurrence of “Text A”), cameras 106, 107, and / or 108 can determine whether the two occurrences of “Text A” appear on the same axis. As shown in Figure 4B, the two occurrences of “Text A” appear on the same axis. Thus, cameras 106, 107, and / or 108 can determine that multiple instances of article 302 are present, and cameras 106, 107, and / or 108 can generate a PASS signal or a FAIL signal as needed. However, one occurrence of pattern “Text B” can indicate that an article that should not be in tray 201 at this stage of the assembly process is placed within tray 201. Thus, cameras 106, 107, and / or 108 can generate a FAIL signal based on the presence of a pattern that is not intended to be present.

[0036] In another aspect, cameras 106, 107, and / or 108 that generate image 400 can determine the presence of pattern "Text B" and can ignore the presence of pattern "Text A" (or any other optional one as needed). In one aspect, one occurrence of pattern "Text B" can indicate the presence of one instance of article 302, and cameras 106, 107, and / or 108 can generate a PASS signal.

[0037] FIG. 5A shows an exemplary image 500 of a tray 201 including one instance of articles 301, 302, and article 501. Cameras 106, 107, and / or 108 that generate the image 500 can be configured to ignore vial caps within region 202 and the presence of the pattern "Text A" within region 203. Instead, cameras 106, 107, and / or 108 that generate the image 400 can determine that there are two other occurrences of a pattern ("Text B") within region 203. In one aspect, the two occurrences of the pattern "Text B" can indicate the presence of one or more instances of article 501, and cameras 106, 107, and / or 108 can generate a PASS signal or a FAIL signal as needed. In another aspect, depending on the pattern configuration on article 501 (e.g., a single instance of article 501 can have one occurrence of "Text B", a double occurrence of "Text B", or a triple occurrence of "Text B"), cameras 106, 107, and / or 108 can determine whether the two occurrences of "Text B" appear on the same axis. As shown in FIG. 5A, the two occurrences of "Text B" appear on the same axis. Thus, cameras 106, 107, and / or 108 can determine that one instance of article 501 is present, and cameras 106, 107, and / or 108 can generate a PASS signal or a FAIL signal as needed. In one aspect, the determination of the axis is used to confirm the presence of any number of articles 501 and generate a PASS signal or a FAIL signal based on the predicted number of instances of article 501 versus the determined number of instances of article 501. FIG. 5B shows an exemplary image 503 of a tray 201 including one instance of articles 301, 302, and article 501. FIG. 5B is similar to FIG. 5A, except that FIG. 5B shows that the pattern "Text B" appears twice along the same axis but at an angle different from the axis of FIG. 5A.

[0038] FIG. 6A shows an exemplary image 600 of a tray 201 that includes two instances of articles 301 and 501. Cameras 106, 107, and / or 108 that generate image 600 can determine that there are four occurrences of the pattern "TEXT B" within region 203. In one aspect, the four occurrences of the pattern "TEXT B" can indicate that one or more instances of article 501 are present, and cameras 106, 107, and / or 108 can generate a PASS signal or a FAIL signal as needed. In another aspect, depending on the pattern configuration on article 501 (e.g., a single instance of article 501 can have one occurrence of "TEXT B", a double occurrence of "TEXT B", a triple occurrence of "TEXT B", or a quadruple occurrence of "TEXT B"), cameras 106, 107, and / or 108 can determine the axis on which the four occurrences of "TEXT B" appear. As shown in FIG. 5A, two occurrences of "TEXT B" appear on a first axis and the other two occurrences of "TEXT B" appear on a second axis. Thus, since the two sets of "TEXT B" appear on different axes, cameras 106, 107, and / or 108 can determine that multiple instances of article 501 are present, and cameras 106, 107, and / or 108 can generate a PASS signal or a FAIL signal as needed. In one aspect, axis determination is used to confirm the presence of any number of articles 501 and generate a PASS signal or a FAIL signal based on the predicted number of instances of article 501 versus the determined number of instances of article 501.

[0039] Figure 6B shows an exemplary image 601 of tray 201 that includes two instances of article 301, article 302, and article 501. Cameras 106, 107, and / or 108 that generate image 601 can determine that there are three occurrences of the pattern "Text B" within region 203. In one aspect, the three occurrences of the pattern "Text B" can indicate the presence of one or more instances of article 501, and cameras 106, 107, and / or 108 can generate a PASS signal or a FAIL signal as needed. In another aspect, depending on the pattern configuration on article 501 (e.g., a single instance of article 501 can have one occurrence of "Text B", a double occurrence of "Text B", a triple occurrence of "Text B", or a quadruple occurrence of "Text B"), cameras 106, 107, and / or 108 can determine the axis on which the three occurrences of "Text B" appear. As shown in Figure 6B, two occurrences of "Text B" appear on a first axis and one occurrence of "Text B" appears on a second axis. Thus, since the two sets of "Text B" appear on different axes, cameras 106, 107, and / or 108 can determine that there are multiple instances of article 501, and cameras 106, 107, and / or 108 can generate a PASS signal or a FAIL signal as needed. In one aspect, the axis determination is used to confirm the presence of any number of article 501s and generate a PASS signal or a FAIL signal based on the predicted number of instances of article 501 versus the determined number of instances of article 501.

[0040] FIG. 7A shows an exemplary image 700 of a tray 201 that includes one instance of article 301, article 302, article 501, article 701, and one instance of article 702. Cameras 106, 107, and / or 108 that generate the image 700 can be configured to ignore vial caps within region 202 and the presence of the patterns "TEXT A" and "TEXT B" within region 203. Instead, cameras 106, 107, and / or 108 that generate the image 700 can determine that there are two separate occurrences of another pattern ("TEXT D") within region 203. In one aspect, the two occurrences of the pattern "TEXT D" can indicate the presence of one or more instances of article 701, and cameras 106, 107, and / or 108 can generate a PASS signal or a FAIL signal as needed. In another aspect, depending on the pattern configuration on article 701 (e.g., a single instance of article 701 can have one occurrence of "TEXT D", a double occurrence of "TEXT D", or a triple occurrence of "TEXT D"), cameras 106, 107, and / or 108 can determine whether the two occurrences of "TEXT D" appear on the same axis. As shown in FIG. 7A, the two occurrences of "TEXT D" appear on the same axis. Thus, cameras 106, 107, and / or 108 can determine that one instance of article 701 is present, and cameras 106, 107, and / or 108 can generate a PASS signal or a FAIL signal as needed. In one aspect, the determination of the axis is used to confirm the presence of any number of articles 701, and a PASS signal or a FAIL signal can be generated based on the predicted number of instances of article 701 versus the determined number of instances of article 701. In the same image 700, cameras 106, 107, and / or 108 can determine that there are two separate occurrences of another pattern ("TEXT C") within region 204.In one aspect, two occurrences of the pattern "Text C" can indicate the presence of one or more instances of article 702, and cameras 106, 107, and / or 108 can generate a PASS signal or a FAIL signal as needed. In another aspect, depending on the pattern configuration on article 702 (e.g., a single instance of article 702 can have one occurrence of "Text C", a double occurrence of "Text C", or a triple occurrence of "Text C"), cameras 106, 107, and / or 108 can determine whether two occurrences of "Text C" appear on the same axis. As shown in FIG. 7A, two occurrences of "Text C" appear on the same axis. Thus, cameras 106, 107, and / or 108 can determine that one instance of article 702 is present, and cameras 106, 107, and / or 108 can generate a PASS signal or a FAIL signal as needed. In one aspect, axis determination is used to confirm the presence of any number of articles 702 and generate a PASS signal or a FAIL signal based on the predicted number of instances of article 702 versus the determined number of instances of article 702. FIG. 7B shows an exemplary image 703 of tray 201 including one instance of article 301, article 302, article 501, article 701, and one instance of article 702. FIG. 7B is similar to FIG. 7A, except that FIG. 7B shows that the pattern "Text D" appears twice along the same axis but at an angle different from the axis of FIG. 7A, and similarly the pattern "Text C" appears twice along the same axis but at an angle different from the axis of FIG. 7A.

[0041] FIG. 8A shows an exemplary image 800 of a tray 201 that includes two instances of articles 301, 302, 501, 701 and one instance of article 702. Cameras 106, 107 and / or 108 that generate the image 800 can determine that there are three occurrences of the pattern "TEXT D" within region 203. In one aspect, the three occurrences of the pattern "TEXT D" can indicate the presence of one or more instances of article 701, and cameras 106, 107 and / or 108 can generate a PASS signal or a FAIL signal as needed. In another aspect, depending on the pattern configuration on article 701 (e.g., a single instance of article 701 can have one occurrence of "TEXT D", a double occurrence of "TEXT D", a triple occurrence of "TEXT D", or a quadruple occurrence of "TEXT D"), cameras 106, 107 and / or 108 can determine the axis on which the three occurrences of "TEXT D" appear. As shown in FIG. 8A, two occurrences of "TEXT D" appear on a first axis and one occurrence of "TEXT D" appears on a second axis. Thus, since the two sets of "TEXT D" appear on different axes, cameras 106, 107 and / or 108 can determine that there are multiple instances of article 701, and cameras 106, 107 and / or 108 can generate a PASS signal or a FAIL signal as needed. In one aspect, the determination of the axis is used to confirm the presence of any number of articles 701 and a PASS signal or a FAIL signal can be generated based on the predicted number of instances of article 701 versus the determined number of instances of article 701. FIG. 8B is similar to FIG. 8A, except that FIG. 8B shows that the pattern "TEXT D" appears twice along a first axis and once along a second axis, where the first and second axes are at a different angle than the axes of FIG. 8A.

[0042] Returning to FIG. 1, cameras 106, 107, and 108 can each independently determine both the presence or absence of one or more patterns in the image and the rotation of each of the one or more patterns in the image of object 104. Cameras 106, 107, and 108 can each perform operations based on the presence or absence and rotation of one or more patterns in the image. Based on the presence of the pattern and the rotation of the pattern, if the camera determines that the correct number of articles is present in the image of object 104, the operation can include sending a PASS signal to pass / fail controller 113. Based on the presence of the pattern and the rotation of the pattern, if the camera determines that an incorrect number of articles is present in the image of object 104, the operation can include sending a FAIL signal to pass / fail controller 113. When cameras 106, 107, and / or 108 each issue a PASS signal to pass / fail controller 113, pass / fail controller 113 can provide a signal to PLC 112 to advance conveyor belt 101 by stepping motor 105 to advance one or more objects 104 and place them under the field of view of the next camera. Pass / fail controller 113 can further send a notification to each of monitors 114 - 116 to display a PASS notification. If one or more of cameras 106, 107, and / or 108 issue a FAIL signal to pass / fail controller 113, pass / fail controller 113 does not provide a signal to PLC 112 to advance stepping motor 105. Pass / fail controller 113 can further send a notification to monitors 114 - 116 associated with the camera that issues the FAIL signal to display a FAIL notification. An operator (e.g., a human or a robot) located at monitors 114 - 116 that display the FAIL notification can perform corrective actions to improve the FAIL state.

[0043] In another aspect, one or more of cameras 106, 107, and 108 can count the number of one or more objects 104. For example, when one or more objects 104 pass through one of cameras 106, 107, and 108, the camera can increment a tally of the one or more objects 104 imaged by the camera. In a further aspect, a plurality of empty spaces may be interspersed among the one or more objects 104. For example, in certain situations, one or more of cameras 106, 107, and 108 may not have an object 104 within the corresponding field of view. Conveyor belt 101 can have a pattern (e.g., a "no tray" pattern) embodied thereon at a position where an object 104 would otherwise be located. Cameras 106, 107, and 108 can identify the pattern and issue a PASS signal to contribute to the advancement of conveyor belt 101.

[0044] Figure 9 shows an exemplary embodiment of a system 100 that illustrates the positioning of cameras 106, 107, and 108 relative to conveyor belt 101. Figure 9 further shows the positioning of monitors 114 - 116. Stepping motor 105 is shown at one end of conveyor belt 101. One or more of PLC 112 and / or pass / fail controller 113 can be housed in housing 901. One or more dispensers 902 can be configured to store one or more articles that are accessed during the assembly of one or more objects. System 100 can include one or more emergency stop (“E-Stop”) buttons 903. The E-Stop button 903 can be actuated at any time to temporarily stop the operation of system 100 for any reason. The E-Stop button 903 can be reset and system 100 can be restarted (e.g., by an operator or technician who has determined that it is safe to do so). System 100 can include one or more opto-switches 904. The opto-switch 904 can be actuated (“tripped”) by placing a finger or thumb on the saddle-shaped structure of the opto-switch 904. This action blocks the optical signal path and causes a switching condition. The opto-switch 904 can be used to accept a visual inspection during “manual trigger” mode and to start / restart the movement of the belt during “autonomous” (or “automatic”) mode.

[0045] System 100 can include a key switch mechanism 905. The key switch mechanism 905 can be used to toggle between an "autonomous" mode and a "manual trigger" mode. Under normal operation, regardless of the mode, the first operator station can include an operator who places a tray on the conveyor belt 101. In one aspect, these trays may be pre-loaded with pre-filled capped vials. In the manual trigger mode, at the second operator station, the operator can place a filter needle tip on the tray. After this operation, the camera 106 inspects the tray for the appropriate articles. At the third operator station, an injection needle tip can be added to the tray. Thereafter, the camera 107 inspects the tray for the appropriate articles. At the fourth operator station, the operator places an empty blister-packaged syringe on the tray. Thereafter, a fifth operator places a Physician Insert (PI) on the tray. After the PI is placed, the camera 108 inspects whether the tray has been fully loaded. When the tray passes through this last station, the full tray exits the conveyor belt 101 for boxing.

[0046] In the automatic mode, the tray automatically moves down along the conveyor belt 101. The system 100 can maintain a residence time (e.g., 1 - 5 seconds) until the conveyor belt 101 moves to the next position. This movement occurs only when all three inspection cameras (e.g., camera 106, camera 107, and camera 108) pass the tray being inspected by the corresponding camera (i.e., "pass"). If a problem occurs at any inspection station, the conveyor belt 101 may enter the "red signal" state (i.e., "fail"), at which point the operator can solve the problem or remove the tray from the conveyor belt 101 (each camera can advance the conveyor belt 101 when the tray is not within its field of view). The advancement of the conveyor belt 101 may depend on all cameras that detect a "pass" tray configuration. The display screens (e.g., monitors 114 - 116) at each camera station can display the video stream of the relevant camera with the "pass", "fail", or "no job" status superimposed according to the inspection results. The online status of the cameras can be reset from monitors 114 - 116 as needed during operation.

[0047] In one aspect shown in FIG. 10, at 1010, a method 1000 is disclosed that includes obtaining a first image of a tray. Method 1000 can include, at 1020, determining the presence or absence of one or more first patterns in the first image. The one or more first patterns can include text patterns, numeric patterns, symbol patterns, and combinations thereof. Method 1000 can include, at 1030, determining the rotation of each of the one or more first patterns in the first image. Method 1000 can include, at 1040, performing an operation based on the presence or absence and rotation of the one or more first patterns in the first image. In one aspect, each step of method 1000 can be performed by a computing device, a camera (having processing capabilities), or a combination thereof. In some aspects, multiple computing devices and / or cameras can be used to perform method 1000. For example, multiple cameras can be used, with a first camera performing steps 1010, 1020, and step 1030 while a second camera performs step 1040. In another aspect, as the tray moves along the assembly line, method 1000 can be repeated at each of several cameras and / or computing devices. For example, steps 1010, 1020, 1030, and 1040 can be performed by a first camera for a particular pattern, and then steps 1010, 1020, 1030, and 1040 can be performed again by a second camera for another particular pattern. Additionally, one or more sub-steps described herein can be performed by a designated camera and / or computing device.

[0048] Determining the presence or absence of one or more first patterns in the first image can include determining the presence of one or two of the one or more first patterns, and determining the rotation of each of the one or more first patterns in the first image can include determining that one or two of the one or more first patterns are on the first axis. Executing an operation based on the presence or absence and rotation of one or more first patterns in the first image can include generating a pass inspection signal and advancing a belt on which a tray is placed. Determining the presence or absence of one or more first patterns in the first image can include determining the presence of three or more of the one or more first patterns. Executing an operation based on the presence or absence and rotation of one or more first patterns in the first image can include generating a fail inspection signal and notifying an operator that a first article associated with the one or more first patterns should be removed from the tray. Determining the presence or absence of one or more first patterns in the first image can include determining the presence of two of the one or more first patterns, and determining the rotation of each of the one or more first patterns in the first image can include determining that two of the one or more first patterns are not on the same axis. Executing an operation based on the presence or absence and rotation of one or more first patterns in the first image can include generating a fail inspection signal and notifying an operator that a first article associated with the one or more first patterns should be removed from the tray.

[0049] Method 1000 can further include obtaining a second image of the tray, determining the presence or absence of one or more second patterns in the second image, determining the rotation of each of the one or more second patterns in the second image, and performing an operation based on the presence or absence and rotation of the one or more second patterns in the second image. The one or more second patterns can include text patterns, numeric patterns, symbol patterns, and combinations thereof. Determining the presence or absence of one or more second patterns in the second image can include determining the presence of one or two of the one or more second patterns, and determining the rotation of each of the one or more second patterns in the second image can include determining that one or two of the one or more second patterns are on a second axis. Performing an operation based on the presence or absence and rotation of the one or more second patterns in the second image can include generating a pass inspection signal and advancing a belt on which the tray is disposed upward. Determining the presence or absence of one or more second patterns in the second image can include determining the presence of three or more of the one or more second patterns. Performing an operation based on the presence or absence and rotation of the one or more second patterns in the second image can include generating a fail inspection signal and notifying an operator that a second article associated with the one or more second patterns should be removed from the tray. Determining the presence or absence of one or more second patterns in the second image can include determining the presence of two of the one or more second patterns, and determining the rotation of each of the one or more second patterns in the second image can include determining that two of the one or more second patterns are not on the same axis.Performing an operation based on the presence or absence and rotation of one or more second patterns in the second image can include generating a failure inspection signal and notifying an operator that a second article associated with the one or more second patterns should be removed from the tray.

[0050] Method 1000 can further include determining a non-slip position in the first image, comparing the determined non-slip position in the first image with a reference position, determining that the determined position is different from the reference position, determining an offset based on the difference between the determined position and the reference position, and sending a signal to a belt controller to adjust the distance and advance the belt on which the tray is placed by the amount of the offset. The offset can be one of a negative value, a positive value, or a zero value. In one aspect, determining an offset based on the difference between the determined position and the reference position, sending a signal to a belt controller to adjust the distance, and advancing the belt on which the tray is placed by the amount of the offset can be performed by one or more cameras. For example, a single camera for determining the offset can be specified. The determination of the offset can be made after each movement of the belt.

[0051] Method 1000 can further include repeatedly acquiring a first image of the tray, determining the presence or absence of one or more first patterns in the first image, determining the rotation of each of the one or more first patterns in the first image, and performing an operation based on the presence or absence and rotation of the one or more first patterns in the first image for each of the plurality of trays.

[0052] Method 1000 can further include counting the number of a plurality of trays, with several empty tray positions scattered among the plurality of trays. Method 1000 can further include counting the number of empty tray positions. Determining the presence or absence of one or more first patterns in the first image can include determining a no-tray pattern. Performing an operation based on the presence or absence and rotation of one or more second patterns in the first image can include advancing a belt on which a no-tray pattern is disposed.

[0053] In an exemplary aspect, the method and system can be implemented on computer 1101 as shown in FIG. 11 and described below. By way of example, camera 106, camera 107, camera 108, PLC 112, and / or pass / fail controller 113 (or components thereof) of FIG. 1 can be computer 1101 as shown in FIG. 11. Similarly, the disclosed method and system can utilize one or more computers to perform one or more functions at one or more locations. FIG. 2 is a block diagram showing an exemplary operating environment 1100 for performing the disclosed method. This exemplary operating environment 1100 is merely an example of an operating environment and is not intended to suggest any limitation as to the scope of use or functionality of an operating environment architecture. Also, operating environment 1100 should not be interpreted as having any dependency or requirement regarding any one or combination of the components shown in exemplary operating environment 1100.

[0054] The present method and system may be operable using a number of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments and / or configurations suitable for use with the present system and method include, but are not limited to, personal computers, server computers, laptop devices and multiprocessor systems. Further examples include set-top boxes, programmable consumer electronics, network PCs, programmable logic controllers (PLCs), minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like.

[0055] The processing of the disclosed method and system can be executed by software components. The disclosed system and method can be described in the general context of computer-executable instructions, such as program modules, being executed by one or more computers or other devices. Generally, program modules include computer code, routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The disclosed method can also be implemented in a grid-based distributed computing environment where tasks are performed by remote processing devices linked through a communications network. In a distributed computing environment, program modules can be located in local and / or remote computer storage media including memory storage devices.

[0056] Furthermore, those skilled in the art will understand that the systems and methods disclosed herein can be implemented via a general-purpose computing device in the form of computer 1101. Computer 1101 can include one or more processors 1103, system memory 1112, and one or more components such as a bus 1113 that couples various components of computer 1101 including one or more processors 1103 to system memory 1112. In the case of multiple processors 1103, the system can utilize parallel computing.

[0057] Bus 1113 can include one or more of several possible types of bus structures, such as a memory bus, a memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of various bus architectures. Bus 1113 and all buses specified in this description can also be implemented via a wired or wireless network connection.

[0058] Computer 1101 typically includes various computer-readable media. Exemplary readable media can be any available media accessible by computer 1101 and is intended to include, but not be limited to, both volatile and non-volatile media, removable and non-removable media. System memory 1112 can include computer-readable media in the form of volatile memory such as random access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). System memory 1112 typically can include data such as image analysis data 1107, and / or program modules such as operating system 1105 and image analysis software 1106 that are accessible to and / or operate by one or more processors 1103.

[0059] In another aspect, computer 1101 can also include other removable / non-removable, volatile / non-volatile computer storage media. Mass storage device 1104 can provide non-volatile storage of computer code, computer-readable instructions, data structures, program modules, and other data for computer 1101. For example, mass storage device 1104 can be a hard disk, a removable magnetic disk, a removable optical disk, a magnetic cassette or other magnetic storage device, a flash memory card, a CD-ROM, a digital versatile disk (DVD) or other optical storage device, a random access memory (RAM), a read-only memory (ROM), an electrically erasable and programmable read-only memory (EEPROM), etc.

[0060] Optionally, any number of program modules, including, by way of example, operating system 1105 and image analysis software 1106, can be stored in mass storage device 1104. One or more (or some combination) of operating system 1105 and image analysis software 1106 can include programming elements and image analysis software 1106. Image analysis data 1107 can also be stored in mass storage device 1104. Image analysis data 1107 can be stored in any one of one or more databases known in the art. Examples of such databases include DB2®, Microsoft® Access, Microsoft® SQL Server, Oracle®, mySQL, PostgreSQL, etc. The database can be centralized or distributed at multiple locations within network 1115.

[0061] In another aspect, a user can input commands and information into computer 1101 via an input device (not shown). Examples of such input devices include, but are not limited to, keyboards, pointing devices (e.g., computer mice, remote controls), microphones, joysticks, scanners, touch-responsive devices such as touchscreens, tactile input devices such as gloves and other body coverings, motion sensors. These and other input devices can be connected to one or more processors 1103 via a human machine interface 1102 coupled to bus 1113, but may also be connected by other interfaces and bus structures such as, but not limited to, parallel ports, game ports, IEEE 1394 ports (also known as FireWire ports), serial ports, network adapter 1108, and / or universal serial bus (USB).

[0062] In yet another aspect, the display device 1111 can also be connected to the bus 1113 via an interface such as the display adapter 1109. It is contemplated that the computer 1101 can have a plurality of display adapters 1109 and the computer 1101 can have a plurality of display devices 1111. For example, the display device 1111 can be a monitor, an LCD (Liquid Crystal Display), a light emitting diode (LED) display, a television, a smart lens, smart glass and / or a projector. In addition to the display device 1111, other output peripheral devices can include components such as a speaker (not shown) and a printer (not shown) that can be connected to the computer 1101 via the input / output interface 1110. Any of the steps and / or results of these methods can be output to an output device in any form. Such output can be in any form of visual representation including, but not limited to, text, graphics, animation, sound, tactile, etc. The display 1111 and the computer 1101 can be part of one device or separate devices.

[0063] In one aspect, the computer 1101 can be coupled to the system 100 via the input / output interface 1110. The computer 1101 can be configured to monitor and store data. The computer 1101 can be configured to store images acquired by a camera connected to the system 100, store data related to pass / fail statistics generated during an inspection occurring in the system, etc. The computer 1101 can also be used as a programming interface to one or more smart devices (such as a smart camera) and / or an embedded logic controller that require customized firmware to operate. The computer 1101 can be used to generate, troubleshoot, upload and store iterations of this software or firmware.

[0064] Computer 1101 can operate in a network environment using logical connections to one or more remote computing devices 1114a, b, c. By way of example, the remote computing devices 1114a, b, c can be personal computers, computing stations (e.g., workstations), portable computers (e.g., laptops, cell phones, tablet devices), smart devices (e.g., smartphones, smartwatches, activity meters, smart apparel, smart accessories), security and / or monitoring devices, servers, routers, network computers, peer devices, edge devices or other common network nodes, etc. The logical connection between the computer 1101 and the remote computing devices 1114a, b, c can be made via a network 1115 such as a local area network (LAN) and / or a common wide area network (WAN). Such network connections may be via a network adapter 1108. The network adapter 1108 can be implemented in both wired and wireless environments. Such network environments are conventional and common in homes, offices, enterprise-scale computer networks, intranets and the Internet. In one aspect, the network adapter 1108 can be configured to supply power to one or more connected devices (e.g., a camera). For example, the network adapter 1108 can comply with the Power over Ethernet (PoE) standard or the like.

[0065] For purposes of illustration, application programs and other executable program components, such as operating system 1105, are shown herein as separate blocks, but such programs and components may reside in different storage components of computing device 1101 at various times and may be executed by one or more processors 1103 of computer 1101. Embodiments of image analysis software 1106 can be stored on or transmitted via some form of computer-readable medium. Any of the disclosed methods can be performed by computer-readable instructions embodied on a computer-readable medium. The computer-readable medium can be any available medium that can be accessed by a computer. By way of example and not limitation, the computer-readable medium can include "computer storage media" and "communication media". "Computer storage media" can include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Exemplary computer storage media can include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer.

[0066] The present method and system can employ artificial intelligence (AI) technologies such as machine learning and iterative learning. Examples of such technologies include, but are not limited to, expert systems, case-based reasoning, Bayesian networks, behavior-based AI, neural networks, fuzzy systems, evolutionary computing methods (e.g., genetic algorithms), swarm intelligence (e.g., ant algorithms), and hybrid intelligent systems (e.g., expert inference rules generated through neural networks or production rules from statistical learning).

[0067] The disclosed method and system have been implemented and tested, and the results were compared with a standard manual-only operator-led assembly line process. The following table shows that the disclosed method and system are superior to a standard manual-only operator-led assembly line process.

[0068]

Table 1

[0069] Unless otherwise specified, no method described in this specification is ever intended to be construed as requiring that its steps be performed in a particular order. Accordingly, method claims are not intended to imply any order in any respect where the steps thereof are not actually recited in the order to be followed, or where it is not otherwise specifically stated in the claims or the specification that the steps are to be limited to a particular order. This holds for any possible implicit basis for interpretation, including logical issues regarding the sequence of steps or the flow of operations, simple meanings derived from grammatical construction or punctuation, the number or types of embodiments described in the specification.

[0070] It will be apparent to those skilled in the art that various modifications and variations can be made without departing from the scope or spirit of the invention. Other embodiments will be apparent to those skilled in the art upon consideration of the specification and practice disclosed herein. The specification and examples are considered exemplary only, and it is intended that the true scope and spirit of the invention be indicated by the following claims.

Claims

1. 1. A method for analyzing an assembly of an object, comprising: acquiring an image of an object disposed at an inspection station, the object comprising a plurality of separate articles; determining a number of occurrences of a pattern corresponding to each of the plurality of articles and an orientation of the pattern; determining a number of each of the plurality of articles in the image based on the determined pattern occurrences and orientations; and comparing the determined number of each of the plurality of items in the image to an expected number of each of the plurality of items in the image; generating a pass or fail signal based on the comparison, the pass signal being generated to indicate that the object is correctly assembled and the fail signal being generated to indicate that the object is incorrectly assembled; A method comprising:

2. The method of claim 1 further comprising: Displaying a notification based on the pass signal or the fail signal. A method comprising:

3. 10. The method of claim 1 , generating a reject signal to cause a machine to take corrective action on said object.

4. 10. The method of claim 1 , generating a reject signal to cause the machine to take corrective action on said object; The method, wherein the corrective action includes adding or removing one or more items to or from the object.

5. The method of claim 1 further comprising: after generating a reject signal, comparing the newly determined number of each of the plurality of articles in the image with an expected number of each of the plurality of articles in the image; generating a pass signal if the newly determined number of each of the plurality of articles in the image matches the predicted number of each of the plurality of articles in the image; A method comprising:

6. The method of claim 1 further comprising: after generating a reject signal, comparing the newly determined number of each of the plurality of articles in the image with an expected number of each of the plurality of articles in the image; generating a pass signal if the newly determined number of each of the plurality of articles in the image matches the predicted number of each of the plurality of articles in the image; displaying a notification based on the pass signal; A method comprising:

7. The method of claim 1 further comprising: after generating a reject signal, comparing the newly determined number of each of the plurality of articles in the image with an expected number of each of the plurality of articles in the image; generating a reject signal if the newly determined number of each of the plurality of articles in the image does not match the expected number of each of the plurality of articles in the image; A method comprising:

8. The method of claim 1 further comprising: after generating a reject signal, comparing the newly determined number of each of the plurality of articles in the image with an expected number of each of the plurality of articles in the image; generating a reject signal if the newly determined number of each of the plurality of articles in the image does not match the expected number of each of the plurality of articles in the image; notifying an operator that the object requires corrective action; A method comprising:

9. The method of claim 1 further comprising: after generating a reject signal, comparing the newly determined number of each of the plurality of articles in the image with an expected number of each of the plurality of articles in the image; generating a reject signal if the newly determined number of each of the plurality of articles in the image does not match the expected number of each of the plurality of articles in the image; notifying an operator that the object requires corrective action; Equipped with The method, wherein the operator is a human or a robot.

10. 10. The method of claim 1 , A method, wherein acquiring an image of an object disposed at an inspection station includes electronically receiving a first image of a first object disposed at a first inspection station.

11. 10. The method of claim 1 , Capturing an image of an object disposed at an inspection station includes electronically receiving a first image of a first object disposed at a first inspection station, the method further comprising: electronically receiving a second image of a second object disposed at a second inspection station remote from the first inspection station, the second object comprising a plurality of separate articles; determining a number of occurrences of a pattern corresponding to each of a plurality of articles in the second image and an orientation of the pattern of the second object; determining a number of each of the plurality of articles in the second image based on the number of occurrences and orientations of the determined object patterns; comparing the determined number of each of the plurality of items in the second image to an expected number of each of the plurality of items in the second image; generating a pass signal or a fail signal based on the comparison, the pass signal being generated to indicate that the second object is correctly assembled and the fail signal being generated to indicate that the second object is incorrectly assembled; A method comprising:

12. 1. A system comprising: An imager; an output device connected to the imager; Equipped with The imager includes: acquiring an image of an object to be placed at an inspection station; configured to analyze the images to determine an assembly of the object; The object is made up of a plurality of separate articles, To determine the assembly of the object, the imager further comprises: determining a number of occurrences of a pattern corresponding to each of the plurality of articles and an orientation of the pattern; determining a number of each of the plurality of articles in the image based on the determined pattern occurrences and orientations; comparing the determined number of each of the plurality of items in the image to an expected number of each of the plurality of items in the image; configured to generate a pass or fail signal based on the comparison; the pass signal is generated to indicate that the object is correctly assembled and the fail signal is generated to indicate that the object is incorrectly assembled; The system, wherein the output device is configured to generate an output based on the pass signal or the fail signal.

13. 13. The system of claim 12, The system, wherein the output generated by the output device is an image or a notification.

14. 13. The system of claim 12, The output generated by the output device based on the fail signal is a red light.

15. 13. The system of claim 12, The system, wherein the output device is a display screen.

16. 13. The system of claim 12, the output device is a display screen; The display screen is configured to display a pass notification based on the pass signal.

17. 13. The system of claim 12, the output device is a display screen; The display screen is configured to display a failure notice based on the failure signal.

18. 13. The system of claim 12, The system, wherein the imager is further configured to process laser, linear, or area imaging.

19. 13. The system of claim 12, The system, wherein the imager is configured to scan, read, decode and apply bar codes.

20. The system of claim 12 , further comprising a processor configured to receive the pass signal or the fail signal.

Images