System and method for ai visual inspection

Abstract

SOLUTION: The invention provides a system and method for visual inspection. The system may be used in a quality assurance station at a manufacturing facility or manufacturing site. The system may evaluate and determine the quality of manufactured or fabricated articles. The system may include a mechanical subsystem for capturing images of the articles. The system may include a sensor such as a camera for capturing data, such as images. The system may include an artificial intelligence system to determine if the articles suffer from impermissible defects.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The following relates generally to visual inspection, and more particularly to systems and methods for visual defect inspection of articles using artificial intelligence. Summary of the Invention

[0002] Visual inspection can be an important part of product manufacturing. Defective products can be costly. Finding defective products at the right stage in the process can be a business critical step to prevent the sale and use of defective goods, determine the root cause associated with the defect, and be able to remedy such cause.

[0003] Existing visual inspection solutions can be difficult to implement where space is limited in manufacturing and other similar settings, and the hardware requirements can increase cost and operational complexity.

[0004] In some cases, visual inspection may still be performed using human inspection. For example, inspection of a camshaft may include a human physically lifting the camshaft and inspecting it with a magnifying tool. Human visual inspection techniques may be limited in accuracy and speed. Human inspection may be prone to overlooking defects. Additionally, human inspection may include a process of manually documenting the visual inspection, which may include documenting identified defects and related information. Such documentation processes may further slow down the inspection.

[0005] With human visual inspection, items deemed defective during the primary inspection may be subjected to secondary and tertiary inspection to confirm the results. These additional inspection steps can significantly increase the inspection time per item.

[0006] For camshaft inspection, a primary inspector can take approximately 45 seconds to 1 minute per camshaft. A secondary inspector can increase the time to approximately 1.5 minutes per camshaft. A tertiary inspector can increase the time further to approximately 2 minutes 15 seconds. These inspection times do not include travel time associated with a human inspector, which can increase the inspection time.

[0007] Many visual inspection solutions in manufacturing that include the use of artificial intelligence (AI) use the cloud. Other solutions may use on-premise servers or standard form factor computer parts. Many users do not want to use the cloud to keep communications locked behind firewalls and create data silos. If visual inspection is implemented on an on-premise server, users will incur significant setup costs to acquire and install specific GPUs for AI inference that may need to be properly optimised. If the AI ​​ingestion pipeline is not properly optimised, many GPUs may be left running without tasks to complete, consuming huge amounts of power while idle. For example, a typical GPU may consume around 250W.

[0008] Furthermore, existing approaches can be difficult to scale up to meet the needs of users, and if the main device fails or is damaged, downtime can occur, which can be costly for the user.

[0009] What is needed is a visual inspection solution that can perform local inference without significant cost, at a level suitable for detecting defects in manufactured articles, and at a speed that meets industrial visual inspection requirements.

[0010] Thus, there is a need for improved systems and methods for visual inspection that overcome at least some of the shortcomings of existing systems and methods.

[0011] A system and method for visual inspection is provided. The system can be used in a quality assurance station at a manufacturing facility or manufacturing site. The system can evaluate and determine the quality of manufactured or processed articles using any one or more of the methods described herein.

[0012] A system for visual inspection of an article is provided, the system including: a camera for acquiring image data of an inspected article, a node computing device for receiving the image data from the camera and analyzing the image data using a defect detection model trained to detect at least one defect type, the defect detection model including a machine learning based object detection model configured to receive the image data as an input and generate defect data as an output describing the detected defects, and a programmable logic controller ("PLC") device for receiving the defect data from the node computing device and determining whether the defect data is acceptable or unacceptable by comparing the defect data to tolerance data.

[0013] The system may further include a robotic arm for autonomously moving the camera to a first imaging position relative to the article under inspection prior to acquiring the inspection image.

[0014] Upon determining that the defect data is unacceptable, the robotic arm may be configured to autonomously move the camera to an item identifier position where the item identifier on the inspected item is within the imaging zone of the camera, and the camera is configured to capture an image of the item identifier.

[0015] The system may further include a robotic arm for autonomously moving the camera through a predetermined sequence of imaging positions during inspection of the article.

[0016] The system may further include an article manipulator configured to engage and rotate the article during inspection.

[0017] The image data may include a plurality of images acquired during rotation of the article, where the node computing device is configured to construct a stitched image from the plurality of images using an image stitching technique, and the image data is provided to the defect detection model as the stitched image.

[0018] The system may further include a user control device for receiving data from at least one of the PLC device and the node device, generating a visualization using the received data, and displaying the visualization on a user interface.

[0019] The user interface may include a PLC mode and a node device mode, where the PLC mode displays a visualization of PLC device operation and the node device mode displays a visualization of node device operation.

[0020] The user controlled device may be configured to automatically switch between PLC mode and node device mode.

[0021] The node computing device may be configured to transmit the defect data to the PLC device upon determining that the detected defect has appeared in a requisite number of consecutive image frames.

[0022] The PLC device may be configured to generate a stop inspection command if it determines that the defect data is unacceptable.

[0023] An artificial intelligence ("AI") visual inspection system is provided that includes a node computing device configured to receive image data of an inspected article and analyze the image data using a defect detection model trained to detect at least one defect type, the defect detection model including a machine learning based object detection model configured to receive the image data as an input and generate defect data as an output describing the detected defects. The AI ​​visual inspection system includes a programmable logic controller ("PLC") device communicatively connected to the node computing device and configured to receive the defect data from the node computing device and determine whether the defect data is acceptable or unacceptable by comparing the defect data to tolerance data.

[0024] The defect data may include a defect class corresponding to the defect type, a defect location, and a confidence level.

[0025] The defect detection model may include a two-stage object detection model.

[0026] A two-stage object detection model may include a region proposal network to generate regions of interest in the first stage, which are sent down the pipeline for object classification and bounding box regression.

[0027] The defect detection model may be configured to perform multi-class classification to classify an instance into one of three or more classes, where the class includes at least two defect types.

[0028] The defect detection model can be modified to operate at the highest possible frame rate.

[0029] The defect detection model may include a neural network that has been optimized by fusing layers of the neural network to compress the size of the neural network to run on the node computing device.

[0030] The node computing device may include a double layer of security where the node device is encrypted and the files on the node device are encrypted.

[0031] The node computing device may include a defect counter for the article that is updated when the PLC device determines that the defect data is unacceptable.

[0032] The node computing device may be further configured to determine whether the detected defect is a true detection by tracking the defect across successive image frames.

[0033] The image data may include a plurality of images acquired during rotation of the article, where the node computing device is configured to construct a stitched image from the plurality of images using an image stitching technique, and the image data is provided to the defect detection model as the stitched image.

[0034] A method for automated visual inspection of an article using artificial intelligence ("AI") is provided. The method may include acquiring image data of an article to be inspected using a camera, providing the image data to a node computing device, analyzing the image data at the node computing device using a defect detection model trained to detect at least one defect type, where the defect detection model includes a machine learning based object detection model configured to receive the image data as an input and generate defect data as an output describing the detected defects, transmitting the defect data from the node computing device to a programmable logic controller ("PLC") device, and determining at the PLC device whether the defect data is acceptable or unacceptable by comparing the defect data to tolerance data.

[0035] The method may further include tracking the detected defects across successive image frames using the node computing device.

[0036] Transmitting the defect data from the node computing device to the PLC device may be performed in response to the detected defect being tracked over a requisite number of consecutive image frames.

[0037] The image data may include a plurality of images acquired during rotation of the article. The method may further include constructing a stitched image from the plurality of images using an image stitching technique and providing the image data to a defect detection model as the stitched image. The method may further include generating a stop inspection command at the PLC device upon determining that the defect data is unacceptable. The method may further include generating an alarm command at the PLC device upon determining that the defect data is unacceptable and sending the alarm command to an alarm system configured to generate and output an alarm.

[0038] The method may further include, upon determining that the defect data is unacceptable, continuing to inspect the article and updating a defect counter for the article to include the detected defects.

[0039] The method may further include, upon determining that the defect data is unacceptable, autonomously moving the camera to an item identifier position where the item identifier on the inspected item is within an imaging zone of the camera, and capturing an image of the item identifier.

[0040] The method may further include linking and storing the item identifier and the defect data in a database.

[0041] Other aspects and features will become apparent to those of ordinary skill in the art upon review of the following description of certain exemplary embodiments.

[0042] The drawings included herein are intended to illustrate various examples of the articles, methods, and apparatus herein. [Brief description of the drawings]

[0043] [Figure 1] FIG. 1 is a schematic diagram of an AI visual inspection system according to one embodiment. [Diagram 2] FIG. 2 is a schematic diagram of an imaging unit of the visual inspection system of FIG. 1 according to one embodiment. [Diagram 3] 1 is a flowchart of a visual inspection sequence according to one embodiment. [Figure 4] 1 is a flowchart of a method for a visual inspection sequence according to one embodiment. [Diagram 5] 2 is a flowchart of a method for implementing the network 156 of node devices 148 of FIG. 1 in real time, according to one embodiment. [Figure 6] 2 is a flowchart of a method of object detection performed by the visual inspection system of FIG. 1 according to one embodiment. [Figure 7] 2 is an exemplary image of an inspected camshaft produced by the visual inspection system of FIG. 1 according to one embodiment. [Figure 8] 2 is an exemplary image of an inspected camshaft produced by the visual inspection system of FIG. 1 according to one embodiment. [Figure 9] 2 is an exemplary image of an inspected camshaft produced by the visual inspection system of FIG. 1 according to one embodiment. [Figure 10] 2 is an exemplary image of an inspected camshaft produced by the visual inspection system of FIG. 1 according to one embodiment. [Figure 11A] FIG. 2 illustrates a first perspective view of a mechanical testing subsystem according to one embodiment. [Figure 11B] FIG. 11B is a second perspective view of the mechanical testing subsystem of FIG. 11A according to one embodiment. [Figure 11C] FIG. 11B is a top view of the mechanical testing subsystem of FIG. 11A according to one embodiment. [Figure 11D] FIG. 11B is a front view of the mechanical testing subsystem of FIG. 11A according to one embodiment. [Figure 11E] FIG. 11B is a rear view of the mechanical testing subsystem of FIG. 11A according to one embodiment. [Figure 11F] FIG. 11B is a side view of the mechanical inspection subsystem of FIG. 11A according to one embodiment. [Figure 11G] FIG. 11B is a front view of the imaging unit of the mechanical inspection subsystem of FIG. 11A according to one embodiment. [Figure 11H] FIG. 11B is a side view of the imaging unit of FIG. 11A according to one embodiment. [Figure 11I] FIG. 11B is a front view of the object holder and object manipulator of the mechanical inspection subsystem of FIG. 11A according to one embodiment. [Figure 11J] 11B is an exemplary bill of materials for the mechanical inspection subsystem of FIG. 11A according to one embodiment. [Figure 12] FIG. 2 is a front view of a mechanical inspection subsystem of the present disclosure, according to one embodiment. [Figure 13] 1 is a flowchart of a method for real-time streaming video analysis, according to one embodiment. [Figure 14A]FIG. 1 illustrates a perspective view of a mechanical inspection subsystem using robotic automation, according to one embodiment. [Figure 14B] FIG. 14B is a top view of the mechanical testing subsystem of FIG. 14A. [Figure 14C] FIG. 14B is a front view of the mechanical testing subsystem of FIG. 14A. [Figure 14D] FIG. 14B is a side view of the mechanical testing subsystem of FIG. 14A. [Figure 15] FIG. 2 is a block diagram illustrating communication between various components of a visual inspection system including a production machine and a camshaft AI visual inspection machine, according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0044] Various devices or processes are described below to provide examples of each claimed embodiment. The embodiments described below are not intended to limit the claimed embodiments, which may include processes or devices different from those described below. The claimed embodiments are not limited to devices or processes having all of the features of any one of the devices or processes described below, or to features common to some or all of the devices described below.

[0045] One or more of the systems described herein may be implemented with computer programs running on programmable computers, each of which includes at least one processor, a data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. For example, and without limitation, the programmable computer may be a programmable logic unit, a mainframe computer, a server, and a personal computer, a cloud-based program or system, a laptop, a personal digital assistant, a mobile phone, a smartphone, or a tablet device.

[0046] Each program is preferably implemented in a high level procedural or object oriented programming language, and / or scripting language, to communicate with a computer system. However, the programs can be implemented in assembly or machine language, if desired. In either case, the language may be a compiled or interpreted language. Each such computer program is preferably stored on a general purpose or special purpose programmable computer readable storage medium or device for configuring and operating the computer when the storage medium or device is read by a computer to perform the procedures described herein.

[0047] A description of an embodiment having multiple components in communication with each other does not imply that all such components are required, rather various optional components are described to illustrate the wide variety of possible embodiments of the present invention.

[0048] Additionally, although process steps, method steps, algorithms, and the like may be described (in this disclosure and / or claims) in a sequential order, such processes, methods, and algorithms may be configured to function in alternative orders. In other words, any sequence or order of steps that may be described does not necessarily indicate a requirement that the steps be performed in that order. Steps of processes described herein may be performed in any order that is practical. Additionally, some steps may be performed simultaneously.

[0049] Where a single device or article is described herein, it will be readily apparent that two or more devices / articles (whether they cooperate or not) may be used in place of the single device / article. Similarly, where two or more devices or articles are described herein (whether they cooperate or not), it will be readily apparent that a single device / article may be used in place of the two or more devices or articles.

[0050] A system and method for visual inspection is provided herein. The system can be used to inspect an article. The system includes a node architecture. The system can perform one-stage or two-stage object detection. A one-stage object detector requires less computing power (compared to a two-stage detector) but provides lower accuracy. A two-stage detector can be expensive in computing resources for higher accuracy, making the introduction of AI in small embedded devices one of the biggest challenges. The system provided herein describes the use of an object detector on a low-power, small computing edge device.

[0051] 1, an exemplary visual inspection system 100 is shown according to one embodiment, configured to perform defect detection using artificial intelligence techniques.

[0052] The visual inspection system 100 is located at an inspection station 108 .

[0053] The inspection station 108 may be a quality assurance station in a manufacturing facility or on a manufacturing site. A quality assurance station may be a station or location on a manufacturing site that is used to evaluate and determine the quality of manufactured or processed articles. A quality assurance station may be a separate station designated to evaluate quality or may be integrated within other parts of the manufacturing or processing process (e.g., on a conveyor, etc.).

[0054] The inspection stations 108 may include automated transport mechanisms, such as conveyor belts, for transporting articles between the inspection stations 108 .

[0055] In an alternative embodiment, system 100 may be advantageously configured to be positioned in-line with existing processes in a manufacturing facility. Inspection station 108 may be a modified conveyor-type system adapted to include components of system 100. In one example, one or more aspects of system 100 (e.g., mechanical inspection subsystem 114, computing system 116, each of which are described below) may be integrated with an automated transport mechanism such that the visual inspection capabilities provided by system 100 may be integrated into existing processes on-site with minimal impact to operations.

[0056] Although the embodiments described herein may refer to the integration and use of system 100 (and other systems described herein) at a manufacturing site or facility, it should be understood that system 100 may be used at any site where visual inspection is performed.

[0057] The visual inspection system 100 is used to inspect an item 110 .

[0058] The system 100 inspects the item 110 and determines whether the item 110 has a defect. The item 110 can be classified by the system 100 as defective or non-defective.

[0059] By identifying the items 110 as defective or non-defective, the inspected items can be differentially processed based on the results of the visual inspection. Defective items 110 can be discarded or otherwise removed from further processing. Non-defective items 110 can continue with further processing.

[0060] Generally, article 110 is an article that is undesirable to have defects. Defects in article 110 may lead to a reduction in the functional performance of article 110 or a larger article (e.g., a system or machine) of which article 110 is a component. Defects in article 110 may reduce the cosmetic appeal of the article. Discovering defective products may be a business-critical process to prevent the sale and use of defective articles, and to determine the root causes associated with the defects so that such causes can be remedied.

[0061] The article 110 may be a processed article. The article 110 may be a manufactured article, which is subject to defects during the manufacturing process. The article 110 may be an article that derives some value from its appearance, and certain defects may adversely affect its appearance. Defects in the article 110 may arise during the manufacturing of the article 110 itself, or during some other process (e.g., shipping, testing).

[0062] Article 110 may be constructed from one or more materials, such as metal, steel, plastic, composites, wood, glass, and the like.

[0063] The articles 110 may be uniform or non-uniform in size and shape.

[0064] The article 110 may include multiple sections. The article sections may be further divided into article subsections. The article sections (or subsections) may be determined based on the appearance or function of the article. The article sections may be determined to facilitate better visual inspection of the article 110 and to better identify unacceptable defective articles.

[0065] The article sections may correspond to different portions of the article 110 having different functions. The different sections may have similar or different dimensions. In some cases, the article 110 includes multiple different section types, and each section type may appear one or more times within the article 110. The sections may be of regular or irregular shape. The different sections may have different defect specifications (i.e., tolerances for certain defects).

[0066] Item 110 may be a stand-alone item intended for use alone, or it may be a component of a larger item (eg, a mechanical part of a larger machine).

[0067] The article 110 may be susceptible to multiple types or classes of defects detectable using the system 100. Example defect types include paint, voids, dents, scratches, blemishes, etc. The defect types may vary depending on the article 110. For example, the defect types may be specific to the article 110 based on the manufacturing process or material composition of the article 110. Defects in the article 110 may arise during the manufacturing itself or through subsequent processing of the article 110.

[0068] Article 110 may be a powertrain part for an automobile and / or motorcycle. The powertrain part may be a camshaft, a differential assembly, a transmission gear, a linkage, a suspension part, or any part or component of the foregoing.

[0069] In a particular embodiment of the system 100, the article 110 is a camshaft. Generally, a camshaft is a shaft to which a cam is fixed or which forms an integral part. The camshaft can be used as a mechanical component in an internal combustion engine. The camshaft opens and closes the intake and exhaust valves of the engine at the correct times and in a precisely defined sequence.

[0070] A camshaft may include multiple different components or parts, each providing a specific function and having specific dimensions. Camshaft components may include any one or more of the following: journals, lobes, ends, and bearings.

[0071] A camshaft can be divided into sections and inspected using system 100. The sections can correspond to different components of the camshaft (e.g., lobe sections, journal sections). The camshaft can have 10-17 sections to be inspected. The 10-17 sections can include lobes and journals.

[0072] System 100 may be configured to acquire in the range of 650-1700 images per camshaft. System 100 may acquire in the range of 50-100 images per section. There may be 13-17 sections per camshaft. For a section, an image may be acquired every 7.2 degrees of camshaft rotation. This may allow for a 360 degree rotation every 2 seconds while giving the AI ​​algorithm enough freedom to adequately track defects in the 50-100 images for each section.

[0073] The camshaft may include a cylindrical rod that runs the length of the cylinder bank with multiple elliptical lobes protruding from the rod. The number of lobes may correspond to the number of valves (e.g., one lobe for each valve). The main journals hold the camshaft in place as it rotates within the engine bay.

[0074] The camshaft may be constructed of any suitable camshaft material. The camshaft may be made from chilled cast iron or billet steel.

[0075] Camshafts may be prone to developing defects, either through the manufacturing process or other processing of the camshaft or its components. Defect types may include paint, voids, dirt, etc. Different components or sections of the camshaft may develop different types of defects. Defects may affect or impact different components differently. This is because different components perform different functions and therefore may have different tolerances for defects. For example, lobe and journal defects may be treated differently and have different tolerances. In some cases, subsections of a component may have different specifications. For example, lobes (which are oval shaped) may have different defect specifications based on the location of the defect on the lobe.

[0076] In some cases, camshafts may have clearance specifications as small as 0.4 mm. The clearance specifications may be location dependent. See Table A below as an example. [Table A]

[0077] Camshaft sections may have non-uniform shapes. This may result in irregular light reflections. Camshafts may be covered with a rust inhibitor (clear). Camshafts may have random dirt or dark oil droplets. In some cases, dirt may cover voids. Defects (e.g. voids, dirt) may have different sizes and shapes.

[0078] The visual inspection system 100 includes a mechanical inspection subsystem 114 and a computing system 116. The mechanical inspection subsystem 114 and the computing system 116 cooperate to inspect the item 110.

[0079] The components of the mechanical inspection subsystem 114 (e.g., actuators, motors, imaging units) may be installed on the conveyor. Some modifications may be made to the components to make them fit on the conveyor. Such a setup may be used, for example, in an article line with an out-conveyor.

[0080] The mechanical inspection subsystem 114 includes an article holder 118 for holding the article 110 prior to inspection. During inspection, the article 110 is lifted from the article holder 118.

[0081] The article holder 118 includes first and second article holders 118a, 118b, which are shown in FIG. 1 as being positioned at or near a first end 119a and a second end 119b, respectively, of the article 110.

[0082] The number and positioning of the item holder 118 (or multiple item holders 118) can vary depending on the items 110 (e.g., shape, size, weight, etc.) or other design choices. Any suitable number of item holders 118 can be used.

[0083] The article holder 118 may be modified as needed to accommodate articles 110 of different shapes, sizes, weights, materials, etc.

[0084] Other variations of the system 100 may not include the article holder 118, for example if the article 110 is inspected by the system 100 while on a conveyor belt or the like. In such cases, the conveyor belt or other feature may act as the article holder 118.

[0085] The mechanical inspection subsystem 114 includes an article manipulator 120 .

[0086] The article manipulator 120 engages and manipulates (i.e., moves) the article 110 to facilitate inspection. The article manipulator 120 manipulates the article 110 in a predetermined manner based on the article being inspected and the imaging setup.

[0087] The article manipulator 120 is adapted to hold the article 110 in a fixed position while the article 110 is inspected by the system 100, and in particular when the article 110 is imaged.

[0088] The article holder 120 can be configured to allow or limit certain types of movement of the article 110 during inspection. For example, the article holder 120 may limit x, y, and z movement of the article 110 while allowing rotational movement 125 of the article 110 along a central axis of the article 110.

[0089] The manner in which the article manipulator 120 moves the article 110 (eg, rotationally, horizontally, vertically, etc.) may depend on various factors, such as the shape of the article 110 and the positioning of the imaging unit.

[0090] Generally, article manipulator 120 manipulates article 110 to position a portion of article 110 within an imaging zone of an imaging unit (e.g., imaging zone 124 of imaging unit 122 described below). In doing so, article manipulator 120 can facilitate imaging and inspection of the entire article 110 (or an entire article section), such as by exposing a previously unexposed section or subsection of article 110 to imaging equipment.

[0091] The article manipulator 120 can be configured to manipulate the article 110 in a series of steps, such as by using a stepper motor.

[0092] The article manipulator 120 includes a first article manipulator 120a and a second article manipulator 120b.

[0093] The first and second article manipulators 120a, 120b may be configured to move along a line of movement 123. The movement of the manipulators 120a, 120b may be driven by a motor.

[0094] Manipulators 120a, 120b move along a line of movement 123 towards article 110 to facilitate engagement with article 110 prior to the start of inspection.

[0095] Movement of manipulators 120a, 120b along line of movement 123 away from article 110 occurs when inspection of article 110 is completed to facilitate disengagement of article 110 and engagement of a new article (if placed).

[0096] The first article manipulator 120a engages a first end 119a of the article 110. The second article manipulator 120b engages a second end 119b of the article 110. The first and second article manipulators 120a, 120b are connected to motors that drive rotational motion of the manipulators 120a, 120b, causing rotation of the article 110 along a line of motion 125.

[0097] Although the embodiment of FIG. 1 shows two article manipulators, any number of article manipulators may be used and may be dictated by the article 110 (size, weight, dimensions) and the type of motion imparted to the article 110.

[0098] In other embodiments, the article manipulator 120 may include a robot or a rotating jig.

[0099] The mechanical inspection system 114 may include an article loader. The article loader is not shown in FIG. 1. The article loader is configured to automatically load and unload the article 110 for inspection. Loading and unloading the article 110 may include placing the article 110 on an article holder 118 and unloading the article from the article holder 118, respectively.

[0100] In one embodiment, the article loader may be a robotic device (e.g., a packaging robot) that can drop the article 110 for inspection, pick up the article 110 after inspection, and deliver the article to either a defective article chute (if a defect exists) or a tote or tray (if no defect exists).

[0101] In some cases, the article loader can automatically load the articles 110 and the articles 110 are removed by a human operator. This may be the case when inspection is the last step before packaging.

[0102] The type of article loader used by the system 100 may depend on the production line and the type of article 110 being inspected.

[0103] The mechanical inspection subsystem 114 also includes an imaging unit 122. The imaging unit 122 captures images of the article 110 (or sections thereof). The captured images are provided to and analyzed by the computing system 116.

[0104] The imaging unit 122 has an imaging zone 124. The imaging zone 124 corresponds to an area for which image data is acquired by the imaging unit 122 and may correspond to a field of view (FOV) of the imaging unit 122.

[0105] In one embodiment, the article 110 is imaged section by section. In such a case, the imaging unit 122 is first positioned where a first section of the article 110 is within the imaging zone 124, and an image of the first section is captured. Then, a second section of the article 110 is positioned within the imaging zone 124 (by moving the imaging unit 122 or the article 110), and an image of the second section is captured. This process can be repeated for successive sections of the article 110 until the entire article 110 has been imaged. In some cases, multiple images of each section may be captured. The multiple images of a section may constitute a sequence of images. For example, multiple images of a first section may be captured, followed by multiple images of a second section. In some cases, the multiple images may represent a 360° imaging of the article 110. In such a case, each of the multiple images includes a portion of the article 110 that has not been previously imaged.

[0106] The imaging unit 122 includes an illumination mechanism 126, a camera 128, and a communications subsystem / interface 130.

[0107] The lighting mechanism 126 illuminates the article 110 with light. The lighting mechanism 126 may focus or confine the illuminated light to the imaging zone 124. The lighting mechanism 126 may be controlled by a lighting controller in communication with the lighting mechanism 126.

[0108] The lighting mechanism 126 can be configured to illuminate homogenous or uniform lighting on the article 110 in the imaging zone 124. The lighting mechanism 126 can also be configured to minimize or eliminate shadows on the article 110.

[0109] The lighting mechanism 126 may use one or more types of lighting, such as tunnel lighting, coaxial lighting, or dome lighting.

[0110] In one embodiment, the lighting arrangement 126 includes tunnel lighting and coaxial lighting, while in another embodiment, the lighting arrangement 126 may include only tunnel lighting.

[0111] The type or types of illumination provided by the illumination mechanism 126 may be selected to provide uniform illumination of the item 110 within the imaging zone 124 and to avoid shadows.

[0112] The camera 128 captures image data of the imaging zone 124, including the item 110 positioned within the imaging zone 124, and stores the image data in memory.

[0113] The camera 128 may be an area scan camera. The area scan camera may be a BASLER area scan camera (e.g., Basler acA1920-40uc). An area scan camera includes a rectangular sensor with two or more lines of pixels exposed simultaneously.

[0114] In one embodiment, the system 100 may have multiple cameras 128. Using multiple cameras 128 can reduce inspection time. Each camera 128 may have its own imaging unit 122. In a particular case, the system 100 may have two cameras, each responsible for imaging half of an article section. In the example of visual inspection of a camshaft, a camera may be positioned at each end of the camshaft and responsible for imaging half of the section (e.g., 8 sections out of a total of 16 sections). Using a two-camera inspection can reduce the overall inspection time of the camshaft by half. Defect detection may be performed as an independent process on the images captured by each camera.

[0115] The camera 128 is connected to the computing system 116 via a communication subsystem 130. The communication subsystem / interface 130 may include a communication interface for the camera 128 and a data transfer connector (e.g., a USB 3.0 cable).

[0116] The communications subsystem 130 transfers data (including the image data generated by the camera 128) to the computing system 116. The data transferred to the computing system 116 includes the image data generated by the camera 128.

[0117] Referring now to FIG. 2, a front view of imaging unit 122 is shown according to one embodiment.

[0118] The imaging unit 122 includes a lighting mechanism 126. The lighting mechanism 126 includes an LED tunnel light 210 and an LED coaxial light 212 positioned above the tunnel light 210. In another embodiment, the tunnel light 210 may be replaced with a dome light. The dome light may be attached to a robotic device, such as a robotic arm (e.g., robotic arm 1484 of FIG. 14).

[0119] The tunnel lighting 210 can be used to provide diffuse lighting to an elongated cylindrical article 110 (e.g., a camshaft). Diffuse lighting can reduce shadows and provide uniform lighting on the article 110.

[0120] Coaxial illumination 212 can be useful for imaging highly reflective articles 110 (e.g., constructed of steel). Light is generated by a diffuse backlight and directed toward the article 110 using a half mirror so that the illumination axis is the same as the optical axis of the camera 128. This can produce a shadow-free image and eliminate reflections.

[0121] The tunnel light 210 is connected to a tunnel light controller 214 via a first lighting cable 220. The tunnel light controller 214 controls the operation of the tunnel light 210, including the amount of light provided.

[0122] The coaxial illuminator 212 is connected to a coaxial illumination controller 220 via a second illumination cable 218. The coaxial illumination controller 220 controls the operation of the coaxial illuminator 212, including the amount of light provided.

[0123] A lens 222 is positioned above the coaxial illumination 212. The lens 222 may be a high resolution lens suitable for installation in a machine vision system. The lens 222 may be a 9 megapixel lens.

[0124] The lens 222 is connected to the camera 128. The camera 128 is an area scan camera. The camera 128 captures image data of the article 110.

[0125] The camera 128 transfers the captured image data to the object detection node device 148 via a communication cable 224. The communication cable 224 may be a USB 3.0 cable.

[0126] 1, the imaging unit 122 is connected to an imaging unit manipulator 132. The imaging unit manipulator 132 moves the imaging unit 122 relative to the article 110 to position the article 110 (or a portion thereof) within the imaging zone 124 of the imaging unit 122.

[0127] The imaging unit manipulator 132 has a frame 134. The frame 134 includes upright supports 136a, 136b and a track 138 fixedly attached to the supports 136a, 136b. As shown in FIG. 1, the track may be disposed substantially parallel to the article 110.

[0128] Imaging unit manipulator 132 includes an imaging unit mount 140 slidably mounted on track 138. Imaging unit mount 140 is configured to connect to imaging unit 122 and secure imaging unit 122 to frame 134 such that imaging unit 122 may be moved along track 138 via mount 140.

[0129] 1, the imaging unit 122 moves horizontally on a track 138 along a line of motion 142 relative to the article 110. In alternative embodiments, the imaging unit manipulator 132 can be configured to move the imaging unit 122 vertically relative to the article 110, or vertically and horizontally relative to the article 110.

[0130] In other embodiments, the imaging unit 122 may be stationary and the article 110 may be moved (eg, via the article manipulator 120 or a conveyor belt) to image different sections of the article 110.

[0131] Positioning the imaging unit manipulator 132 so that the imaging unit 122 can move in the horizontal direction 142 relative to the article 110 may enable the imaging unit 122 to suitably capture image data for multiple sections of the article 110 along the horizontal axis.

[0132] The imaging unit manipulator 132 can work in coordination with the article manipulator 120 to capture an image of the entire article.

[0133] The imaging unit manipulator 132 also includes motors for moving the imaging unit 122. The motors may be stepper motors. In one embodiment, the imaging unit manipulator 132 includes an actuator for moving the imaging unit 122 vertically (up and down) and a stepper motor for moving the imaging unit 122 horizontally along the length of the article 110. The actuators may be pneumatic cylinders.

[0134] In another embodiment of the system 100, the mechanical inspection subsystem 114 includes a robotic system for imaging the article 110. The robotic system may include a robotic arm and a robotic arm controller including a computing device. In one embodiment, the robotic system may be the robotic system 1482 of FIG. 14. The imaging unit 122 (including the camera 128 and the lighting mechanism 126) may be mounted or otherwise attached to the robotic arm. The robotic arm may be replaced by any one or more of the imaging unit manipulator 132, the frame 134, the upright support 136, the track 138, and the imaging unit mount 140. The robotic arm may be capable of moving in three dimensions (i.e., not strictly along the line of movement 142). In general, the robotic arm is used to move the imaging unit 122 (the camera 128 and the lighting mechanism 126) relative to the article 110 to be inspected and capture images of multiple sections of the article, such as along the length of the article 110. The robotic arm may function autonomously (i.e., perform movements such as moving the camera / imaging unit autonomously). For example, the movements of the robotic arm to be performed during imaging of the article 110 may be programmed into the robotic arm controller. Due to different shapes and sizes of the article, the programmed movements may vary. The movements of the robotic arm to image the article 110 may include a series of movements performed for each article 110. The robotic system may include a communication interface such as an Ethernet / IP module (e.g., Ethernet / IP module 1512 of FIG. 15) for communicating with other system 100 components. For example, the robotic system may use an Ethernet / IP module to communicate with a PLC of a production machine or a PLC 146 of the visual inspection system 100.

[0135] The visual inspection system 100 includes a computing system 116 for performing various control, analysis, and visualization functions of the system 100. The computing system 116 is communicatively coupled to the mechanical inspection subsystem 114 to facilitate data transfer between the computing system 116 and the mechanical inspection subsystem 114 to perform the visual inspection.

[0136] The computing system 116 includes a programmable logic controller (PLC) 146, a node device 148, and a user control device 150. The devices 146, 148, 150 are communicatively connected to one another, which may include wired or wireless connections. The devices 146, 148, 150 may be connected to one another via a network 152, such as a wired or wireless network. The network 152 may include direct wired connections between components described herein. Communication of aspects of the mechanical inspection subsystem 114 with other components, such as the PLC 146, via the network 152 is represented by a solid line from the mechanical inspection system 114 to the network 152.

[0137] The PLC 146 is a digital computer adapted to automate the electromechanical processes of the visual inspection system 100. The PLC 146 can provide high reliability control, as well as ease of programming and process fault diagnosis.

[0138] The PLC 146 includes a memory (e.g., a RAM or a flash memory) for storing and a processor for executing a PLC computer program configured to control the various components and operations of the visual inspection system 100. The PLC 146 includes a communication interface for transmitting data (e.g., defect data) to and receiving data from other components of the system 100.

[0139] The PLC computer program contains computer-executable instructions for controlling the movement and operation of one or more components of the mechanical inspection subsystem 114, interfacing with the node device 148 and the user control device 150, and processing defect data received from the node device 148.

[0140] The PLC 146 is also configured to execute a test sequence, such as the test sequence 400 of FIG.

[0141] The PLC 146 controls, via a PLC computer program, a number of actuators and motors that move components of the mechanical testing subsystem 114. Any suitable number of actuators and motors may be used.

[0142] The PLC 146 can control the actuators and stepper motors in either manual or automatic mode.

[0143] The actuator may be a cylinder, such as a pneumatic cylinder.

[0144] The motor may be a stepper motor.

[0145] The system 100 includes three actuators, including a first actuator, a second actuator, and a third actuator.

[0146] The first actuator is for horizontal movement. In response to commands from the PLC 146, the first actuator moves the article manipulator 120a horizontally toward and away from the first end 119a of the article.

[0147] The second actuator is for horizontal movement. In response to commands from the PLC, the second actuator moves the article manipulator 120b (which includes a stepper motor) horizontally toward and away from the second end 119b of the article 110.

[0148] The third actuator is for vertical movement. The third actuator moves the imaging unit 122 up and down relative to the article 110 under commands from the PLC 146.

[0149] The mechanical testing subsystem 114 also includes two stepper motors, including a horizontal translation motor and a rotary translation motor.

[0150] The horizontal movement motor moves the imaging unit 122 horizontally along the track 138 under command from the PLC 146. The imaging unit 122 can be moved in steps depending on the imaging position of the imaging unit 122. The imaging position can correspond to a section of the article 110. The imaging position can be predetermined. Moving to a subsequent step or position can be based on the PLC 146 receiving a signal from another component of the system 100 (e.g., from the node device 148) or using event-based movement. For example, a robot or camera can be instructed to move from a first section of the article 110 to a second section of the article 110 based on an event, such as detecting that the first section is complete.

[0151] Upon command from the PLC 146, the rotary translation motor rotates the article 110 to facilitate inspection of the entire article (e.g., a full 360°). The article 110 can be rotated in steps (e.g., 4 degrees) corresponding to subsections of the article 110.

[0152] The imaging unit 122 and the article manipulator 120a, 120b each have a return / home position and an advance position. To return to the home position, the article manipulator 120a is moved horizontally away (to the left) from the article 110 by the first actuator. To return to the home position, the article manipulator 120b is moved horizontally away (to the right) from the article 110 by the second actuator. To return to the home position, the imaging unit 122 is moved vertically upward away from the article 110 by the third actuator. To advance to the advance position, the article manipulator 120a is moved horizontally (to the right) towards the article 110 by the first actuator. To advance to the advance position, the article manipulator 120b is moved horizontally (to the left) towards the article 110 by the second actuator. To advance to the advance position, the imaging unit 122 is moved vertically downward towards the article 110 by the third actuator. The imaging unit 122 also moves horizontally (eg, from right to left) down the camshaft to scan the camshaft via a horizontal translation motor.

[0153] The PLC 146 is communicatively connected to a number of sensors configured to provide machine status data to the PLC 146. The machine status data provides status information regarding components of the subsystem 114, such as actuators, and whether or not an item 110 is loaded.

[0154] The PLC 146 interfaces with the node device 148 via a PLC computer program. The PLC 146 can communicate with the node device 148 via the TCP communication protocol.

[0155] The data that the PLC 146 receives from the node device 148 includes a code that indicates whether a fault has been detected in the node device 148 .

[0156] For example, if the node device 148 discovers a defect, the PLC 146 may receive an NG code from the node device 148 indicating that a defect has been discovered in the item 110 and that the item 110 is defective (i.e., “bad”).

[0157] The NG code may include defect data that describes the defect, which may include the defect type or class (e.g., void, dirt, paint, scratch, dent, etc.) and the defect size.

[0158] If the node device 148 does not find any defects, the PLC may receive an OK code from the node device 148 indicating that no defects were found.

[0159] The PLC 146 processes the defect data received from the node device 148 via a PLC computer program. The PLC 146 receives motor data from the stepper motor. The motor data may include a horizontal reading (e.g., from a first stepper motor horizontal) that identifies a section of the article 110. The motor data may include a rotation reading (from a second stepper motor rotation) that identifies a camshaft angle. The camshaft angle may be referenced to a pin / keyway. The PLC 146 uses the motor data to identify the exact X and θ coordinates of the article 110 (the camshaft). The motor data includes section data (e.g., an x ​​coordinate value) and angle data (e.g., a θ coordinate value). The section data describes a particular section of the article 110 (e.g., may be associated with a horizontal coordinate value x). The angle data describes an angle of the particular article (with respect to the article manipulator 120 (e.g., article manipulator 120b includes a stepper motor and maintains the angle of the article 110)). The section data and angle data together describe a subsection of the article 110. An article subsection has a particular defect specification (i.e., for a particular section / angle combination) that includes a tolerance range for a particular defect type. The defect specification, including the tolerance range, is stored in memory by the PLC 146 and linked to a particular section and angle combination. Different subsections of an article may have different tolerance ranges.

[0160] As mentioned above, the PLC 146 is also configured to transmit data to the node device 148 .

[0161] The data transmitted from the PLC 146 to the node device 148 may include an inspection code. The inspection code instructs the node device 148 to start detection. For example, the article 110 may be rotated slowly in steps by a rotating motor. For each step, the PLC 146 can send an inspection code to the node device 148 instructing the node device 148 to start detection.

[0162] In another embodiment, the computing system 116 can use an image stitching technique. For example, the PLC 146 instructs the node device 148 to take an image of the article 110. The PLC 146 then rotationally moves the article 110 (i.e., rotates the article 110 incrementally) and instructs the node device 148 to obtain another image of the article 110 (i.e., at least a portion of the second image is a previously unseen portion of the article 110). This process of operations by the PLC 146 and the node device 148 can be repeated until the entire 360° of the article 110 is imaged. The multiple images representing the 360° image of the article 110 are then stitched together by the node device 148 to construct a stitched image according to an image stitching technique. The node device 148 then performs object detection on the stitched image. In such a case, the object detection can be performed once by the node device 148 at the last rotation of 360° on the stitched image. Upon completion of a 360 degree rotation of the article 110, the PLC 146 may send an inspection code to the node device 148 instructing the node device 148 to begin detection. The node device 148 may construct a stitched image for multiple adjacent sections of the article 110 and perform object detection on the stitched image. Image stitching may advantageously allow inference to be performed only once on the stitched image that represents the entire section of the article covered by the stitched image. The act of cropping the images together into a final large (stitched) image may provide the system with the highest quality image.

[0163] The data sent from the PLC 146 to the node device 148 may include an end code. The end code instructs the node device 148 to end detection. The end code may instruct the node device 148 to end detection until a new item 110 is placed.

[0164] The PLC 146 may also send handshake data to and receive handshake data from the node device 148. The handshake data includes specific codes that, when exchanged between the PLC 146 and the node device 148, establish a communication channel between the devices 146, 148.

[0165] PLC 146 interfaces with user control device 150 via a PLC computer program. PLC 146 can transmit data to user control device 150 indicative of the status of operation of subsystems 114. Such received data can be visualized by user control device 150. PLC 146 can also receive data from user control device 150. Such data can control operation of PLC 146 or subsystems 114 according to user input received at user control device 150.

[0166] Referring now to node device 148, node device 148 may be an edge computing device. Node device 148 may be an edge device that is specifically configured to run AI applications and perform deep learning tasks, and may include processing components suitable or adapted to perform such tasks.

[0167] In one embodiment, the node device 148 is a JETSON XAVIER device.

[0168] The node device 148 may provide an edge solution that is secure and does not require connection to devices outside the inspection site or facility, allowing data to remain on-premise.

[0169] The node device 148 may be encrypted (eg, device level encryption) and files on the node device 148 may be encrypted (eg, file level encryption). This may provide a double layer of security.

[0170] In one embodiment, the system 100 may include multiple node devices 148 arranged in a node architecture. The multiple node devices 148 may operate simultaneously and communicate back and forth with the PLC 146. Such an architecture may facilitate efficient scale-up for users. The node architecture may also reduce potential downtime caused by equipment issues, because if one node device 148 fails or is damaged, the non-functioning node device may be replaced by plugging in an alternative node device.

[0171] The node device 148 can advantageously reduce costs associated with visual inspection systems, especially in comparison to existing edge-based approaches that may typically use regular Nvidia or equivalent graphics cards that require computer hardware to run.

[0172] By using the node device 148 to perform inference tasks (i.e., object detection, defect classification), the visual inspection system 100 can reduce power consumption compared to existing on-site inference approaches that use large servers with GPUs that may have consumption in the range of 250 W. For example, the node device 148 may have a significantly lower consumption in the range of 30 W.

[0173] The node device 148 is an embedded computing device.

[0174] The node device 148 includes processing components and storage components.

[0175] The processing components may include one or more CPUs and GPUs.

[0176] The storage components include memory such as RAM and flash.

[0177] The node device 148 also includes communication interfaces for communicating with the PLC 146, the user control device 150, and the camera 128.

[0178] The node device 148 stores in memory one or more computer programs including computer-executable instructions that, when executed by a processing component of the node device 148, cause the node device 148 to perform the actions and functions described herein.

[0179] The node device 148 is configured to perform artificial intelligence at the edge. Advantages of performing AI at the edge using the node device 148 include faster inference speed, lower latency between image capture and inference results, and no need for data to leave the facility.

[0180] The node device 148 is configured to send and receive specific codes to and from the PLC 146 to establish communications.

[0181] The node device 148 receives the check code from the PLC 146. Upon receiving the check code, the node device 148 starts the object detection pipeline.

[0182] The node device 148 is communicatively connected to the camera 128 and instructs the camera 128 to capture images of the item 110. The node device 148 may communicate with the camera 128 via an API or the like.

[0183] The node device 148 includes a defect detection model 154. The defect detection model 154 is a machine learning based model configured to analyze image data of the article 110 and identify defects therein. The defect detection model 154 uses object detection techniques to identify defects in the image data. The defect detection model 154 may be configured to perform multi-class classification to classify instances into one of three or more classes. The classes may correspond to different defect types that the defect detection model 154 is trained to detect. The classes may include at least two defect types.

[0184] The node device 148 may include software modules configured to train, optimize, and execute the fault detection model 154.

[0185] The defect detection model 154 can utilize an object detection API, such as the TensorFlow Object Detection API, which can be an open source framework that facilitates building, training, and deploying object detection models.

[0186] The defect detection model 154 is configured to perform defect classification. Defect classification involves assigning detected defects to defect classes. The defect classes are associated with particular defect types (e.g., dents, paint, scratches, etc.) that the defect detection model 154 is trained to detect. The object detection model 154 can be optimized (i.e., through training of the model 154) to learn how to ignore anomalies such as dirt, smudges, oil and water marks that may be present on the article 110. The ability of the object detection model 154 to ignore such anomalies can advantageously reduce the rate of false alarms.

[0187] The defect detection model 154 analyzes the image data to generate defect data, which includes information about the identified defects, such as the defect type, the defect size, the defect confidence level, and the defect location.

[0188] The node device 148 stores the defect data in one or more storage components. The node device 148 may transmit the defect data to the PLC 146 for further processing. The PLC 146 processes the defect data and determines whether the defect is an unacceptable defect (outside of a tolerance range) or an acceptable defect (within a tolerance range).

[0189] The node device 148 may transmit the defect data to the user control device 150 for visualization.

[0190] In one embodiment, the defect detection model 154 includes a neural network 156. The neural network 156 is trained on images of defective and non-defective articles 110. Training may be performed according to known training techniques and those described herein.

[0191] In one embodiment, neural network 156 is generated using model training and transformation for inference, which may include taking a pre-trained model, training the pre-trained model with new data, pruning the model, re-training the pruned model to generate an output model, and running the output model through an engine (e.g., the TensorRT engine and / or an optimized CUDA backend).

[0192] The neural network 156 may be a Faster Region-based Convolutional Neural Network (faster R-CNN) or a CSPDarknet53.

[0193] Faster R-CNN may have a ResNet or Inception classification backbone. ResNet may be ResNet50 or ResNet101. Inception may be Inception v2.

[0194] Neural networks 156 (e.g., CSPDarknet53) can achieve similar results to two-stage object detection while being only a one-stage detector, using new state-of-the-art techniques such as weighted residual combination (WRC), cross-stage partial combination (CSP), cross mini-batch normalization (CmBN), self-adversarial training (SAT), and Misch activation.

[0195] The neural network 156 described above may be optimized to run on the node device 148. The optimization may be performed according to the optimization methods described herein. The optimization may include any one or more of Kernel Auto-Tuning, Layer and Tensor Fusion, Precision Calibration (FP32-FP16), CUDA optimization, and utilizing Dynamic Tensor memory. The optimization may fuse layers of the neural network 156 to compress the network size for execution on the node device 148. This may enable the network 156 to run efficiently on the node device 148, enabling local inference at the edge.

[0196] The node device 148 may have a per-image processing time, enabling real-time speeds. The node device 148 may have a processing time of approximately 25 ms to 60 ms per image for image sizes greater than 1000 (width) by 1000 (height) pixels. In embodiments where the node device 148 is a typical GPU, the processing time per image may be in the range of 5 ms to 60 ms.

[0197] The defect detection model 154 may include a two-stage object detection model. The two-stage object detection model may be trained using a training framework such as TensorFlow. The two-stage object detection model may include modifications to any one or more of the input image size, batch number, and first stage and second stage proposals. Such modifications may improve the efficiency of the model while maintaining accuracy as high as possible or at an acceptable level.

[0198] A two-stage detection model can use a region proposal network to generate regions of interest in the first stage, which are then sent down the pipeline for object classification and bounding box regression.

[0199] Once the defect detection model 154 is pre-trained, the weights of the neural network 156 can be fine-tuned using transfer learning, which can reduce the size of the data set and shorten the total training time while still obtaining acceptable results.

[0200] The defect detection model 154 may include a CNN that is pre-trained on a very large dataset (e.g., ImageNet, which contains 1.2 million images with 1000 categories, or Common Objects in Context (COCO)), and then the CNN may be used as an initialization or fixed feature extractor for the defect detection task.

[0201] Fine-tuning using transfer learning may include fine-tuning the CNN. Fine-tuning the CNN may include replacing and retraining the classifier on the CNN on the new dataset and fine-tuning the weights of the pre-trained network by continuing backpropagation. All of the layers of the CNN may be fine-tuned, or some of the early layers may remain fixed (e.g., due to concerns about overfitting) and some higher level parts of the network may be fine-tuned. This may be motivated by the observation that early features of a CNN include more general features (e.g., edge detectors or color blob detectors) that are useful for many tasks, but later layers of the CNN become increasingly more specific to the details of the classes contained in the original dataset.

[0202] In one embodiment, the fault detection model 154 (e.g., neural network 156) may be modified to run more efficiently on the node device 148 (e.g., Jetson Xavier).

[0203] For example, the model may be modified to run at the highest possible frame rate (FPS), which is the frame rate of inference per image. The frame rate may be determined by a number of factors (e.g., image size, number of layers, number of anchor boxes, algorithms in the network, operations in the network, floating point precision, layer fusion, etc.). Higher FPS inference by the node device 148 can reduce inspection time to a level more suitable for industrial manufacturing inspection processes.

[0204] The fault detection model 154 may be optimized for inference on the node device 148.

[0205] The defect detection model 154 can be optimized with TensorRT or OpenCV (CUDA). The optimization can be done after an optimal number of training iterations to ensure that the model is not overfitted, and after training with augmented images to reduce overfitting.

[0206] Training of the defect detection model 154 may include image augmentation. Data augmentation can be performed to increase the variability of the input images, so that the designed object detection model has higher robustness to images acquired from different environments. Two augmentation strategies that may be implemented include photometric distortion and geometric distortion in aiding in training the object detection model. For photometric distortion, the brightness, contrast, hue, saturation, and noise of the image can be adjusted. For geometric distortion, random scaling, cropping, flipping, and rotation can be added. Other special augmentations include, for example, random erasure, CuTOUT, Dropout / Drop Connect / Dropblock in feature maps, Mixup (with two different images), CutMix, and mosaic augmentation.

[0207] By augmenting the images during training, the training set can be artificially grown, which can increase the robustness of the generalization ability of the defect detection model 154 (object detector).

[0208] The network can then be run on the node device 148 in Python with FP16 precision. Network optimizations are performed on the node device 148 itself to take advantage of available hardware and kernels that are CUDA optimized at the current run time.

[0209] The user control device 150 includes a processor, a memory, and an output interface, including a display 158. The user control device 150 includes a communication interface for transmitting data to and receiving data from other components of the system 100, such as the PLC 146 and the node device 148.

[0210] The user control device 150 also includes an input interface for receiving input from an operator. The input interface may be a touch screen on the display 158.

[0211] In one embodiment, the display 158 of the user control device 150 can operate as both a touch display and a high-resolution regular monitor.

[0212] The memory stores data received from the PLC 146 and the node device 148, including the defect data.

[0213] The memory also stores one or more computer programs for generating and displaying a user interface 160. The user interface 160 includes a visualization 162 of the results of the visual inspection.

[0214] The processor may generate the visualization 162 based on the defect data received from the node device 148.

[0215] In one embodiment, the user interface 160 may include a PLC mode and a node device mode. In the PLC mode, the user interface 160 generates and displays visualizations of PLC operations (e.g., the mechanical inspection subsystem 114 in operation). The visualizations in the PLC mode are generated using data from the PLC 146. In the node device mode, the user interface 158 generates and displays visualizations of node device operations (e.g., object detection). The visualizations in the node device mode are generated using data from the node device 148. Switching between the PLC mode and the node device mode may be programmed to occur automatically (e.g., to the node device mode when detection is being performed) or may be performed manually, such as by receiving a command from an operator via an input interface.

[0216] In one embodiment, the visual inspection system 100 can be configured to stop inspection of the article 110 when a defect is detected (and confirmed). Such an embodiment can provide a speed advantage by limiting unnecessary inspection of the defective article 110. For example, if the article 110 has ten sections and a defect is identified by the computing system 116 in the third section of the article 110, the visual inspection system 100 may stop inspecting the article and not proceed to inspect the remaining seven sections. The article 110 is removed and a new article is placed and inspected. In some cases, when stopping inspection of the article 110, the system 100 can read an article identifier on the article 110 (e.g., by moving the imaging unit 122 and acquiring an image of the article identifier) ​​and associate the identifier with the defective article before removing it. Stopping the inspection can be initiated by the PLC 146 (e.g., after determining that the detected defect is unacceptable). The PLC 146 can send an inspection stop command to one or more components of the mechanical inspection subsystem 114. When the inspection stop command is received and processed by the receiving component, it can cause the receiving component to perform one or more actions that facilitate stopping the inspection (e.g., moving the imaging unit 122 to a different position, removing the item 110 from the item manipulator 120, moving the item 110 from the inspection area, placing a new item on the item holder 118, etc.).

[0217] In another embodiment, the visual inspection system 100 may continue after defects are found (i.e., does not stop after the first defect is identified and the article is determined to be defective). In such a case, the computing system 116 may include a defect counter for the article 110, which is updated each time a defect is found (i.e., detected in the image data) on the article 110. The defect counter is configured to keep a record of defects detected for the inspected article. Updating the defect counter may include adding defect data for the newly detected defect to the record. The defect counter may include one or more software modules and may be implemented by the node device 148. In some cases, the defect counter may be updated with a new defect only when the defect is confirmed as a true detection using object tracking. The defect counter may be updated with a new defect only if the defect is determined to be unacceptable by the PLC 146 (e.g., by comparison to a threshold value).

[0218] The visual inspection system 100 may provide faster and / or more accurate visual inspection of the article 110 as compared to existing techniques.

[0219] In some cases, visual inspection system 100 may be able to detect defects (e.g., voids in a camshaft) in the presence of other miscellaneous objects (e.g., dirt) that would normally obscure such defects (e.g., when inspected by a human). This may be particularly advantageous in manufacturing and other environments where article 110 is not always clean and may have materials on article 110 that may prevent detection under existing techniques.

[0220] An embodiment can inspect subsections within each section at 4 degrees each, with 360 degrees of rotation for each section (rotation provided by article manipulator 120). For each section of article 110 (e.g., a camshaft), article 110 can be imaged, rotated a certain number of degrees, and imaged again. This is performed for a full rotation of article 110 for each section.

[0221] In one embodiment, the defect data generated by the computing system 116 may be stored in a defect analysis database. The defect data stored in the analysis database may be provided from the node device 148. The defect data in the defect analysis database may be analyzed for analytical purposes to provide a plant manager or other operator with a larger picture of defect detection in the items. For example, defect data for multiple items 110 may be provided to a computing device configured to analyze the defect data and identify trends, sizes, etc. The analysis may include using statistical libraries and machine learning (e.g., clustering) to look at defect types and other trends. A report containing such information may be generated and displayed to an operator, for example, via the display 158.

[0222] 3, there is shown a visual inspection sequence 300 according to one embodiment. The sequence 300 may be performed by the system 100 of FIG.

[0223] At 310 the sequence begins.

[0224] At 312 , the article 110 is placed on the article holder 118 .

[0225] At 314, the article manipulator 120 engages the article 110. This allows the article manipulator 120 to move (e.g., rotate) the article 110 during inspection.

[0226] At 316, the imaging unit 122 is moved to a first imaging position. The imaging unit 122 is moved via the imaging unit manipulator 132.

[0227] At 318, the camera 128 of the imaging unit 122 captures an image of the item 110 at the current imaging position.

[0228] At 320, the image is transmitted from the camera 128 to the node device 148.

[0229] At 322, the node device 148 performs object detection on the received images to detect defects in the article 110. The object detection is performed using the defect detection model 154. The images are analyzed to generate defect data.

[0230] At 324 , the image defect data is sent to the PLC 146 .

[0231] At 326, the PLC determines whether the defect data is within a predetermined tolerance range. The predetermined tolerance range is stored as tolerance range data in the PLC 146. The determination at 326 may include comparing or referencing some or all of the defect data to the tolerance range data.

[0232] At 328, if the defect data is not within the predetermined tolerances, the item 110 is identified as defective and the inspection is terminated.

[0233] At 330, the imaging unit 122 is returned to the starting or home position via the imaging unit manipulator 132. Upon returning to the home position, the sequence can return to begin the process at 10 for another article 110.

[0234] At 332, if the defect data is within the predetermined threshold at 326, the PLC 146 determines whether there is another section of the article 110 to be imaged.

[0235] At 334, if there is another section to be imaged, the imaging unit 122 is moved, via the imaging unit manipulator 132, to a second imaging position corresponding to the section not being imaged.

[0236] Once the imaging unit 122 has been moved to the second imaging position, the process 300 returns to 318 where an image of the item 110 is captured at the current imaging position (ie, the second imaging position) and the process begins again.

[0237] If there are no more sections of the article 110 to be imaged, the inspection ends at 328. At 330, the imaging unit 122 is returned to the home position so that a new inspection can begin at 310.

[0238] 4, there is shown a visual inspection sequence 400 according to one embodiment. The inspection sequence 400 may be performed by the visual inspection system 100 of FIG.

[0239] At 410, the test sequence 400 begins.

[0240] At 412, the PLC 146 establishes communication with the TCP port, which may occur upon start-up of the inspection system 100.

[0241] At 414, a handshake is performed between the PLC 146 and the node device 148 by sending and receiving specific codes. The handshake initiates communication between the PLC 146 and the node device 148. The handshake is initiated when one device sends a message to another device indicating that it wants to establish a communication channel. The two devices 146, 148 send messages back and forth that allow them to agree on a communication protocol.

[0242] At 416, after the handshake is complete, the node device 148 continues to listen to the PLC 146 to receive the inspection code. The inspection code instructs the node device 148 to perform an inspection of the item 110 within the imaging zone 124 of the camera 128.

[0243] At 418, the article 110 is placed on the article holder 118. The article 110 can be placed manually or automatically. Automatic loading can include integration with a conveyor or the like.

[0244] At 420, the first article manipulator 120a advances and moves toward the article 110. The first article manipulator 120a is moved via the first actuator (center 1).

[0245] At 422, the second article manipulator 120b advances and moves toward the article 110. The second article manipulator 120b is moved via a second actuator (center 2).

[0246] At 424, the rotary translation motor (motor 2) of the second article manipulator 120b rotates to engage a pin in the article 110 with a keyway (eg, item 14 in FIG. 11I).

[0247] At 426, the camera 128 is moved down until the camera 128 reaches a first inspection position and then moved to the left.

[0248] At 428, the rotational movement motor (motor 2) of the second article manipulator 120b begins to move the article 110 stepwise. After each step, an inspection code is sent to the node device 148 and object detection is initiated. In other embodiments, the inspection code may be sent to the node device 148 upon completion of a predetermined number of rotational steps (e.g., upon completion of a 360° rotation). In one such embodiment, object detection may be performed on a stitched image constructed from multiple images corresponding to the steps.

[0249] At 430, an image of the article 110 is captured. The image is analyzed by the node device 148 using AI algorithms to detect defects in the image (e.g., via the defect detection model 154).

[0250] At 432, the rotary translation motor (Motor 2) of the second article manipulator 120b continues to rotate the article 110 360 degrees while the article 110 is imaged by the camera 128.

[0251] At 434, the camera 128 is moved to a second inspection position and the rotary translation motor (motor 2) of the second article manipulator 120b rotates the article 110 another 360°. In one embodiment, the camera 128 (imaging unit 122) is moved via a stepper motor. In another embodiment, the camera 128 (imaging unit 122) is moved using a robotic arm (e.g., robotic arm 1484 of FIG. 14).

[0252] At 436, inspection continues, moving to successive inspection positions to perform inspections until each section of the article 110 has been inspected.

[0253] At 438, the PLC 146 sends an end code to the node device 148. The end code instructs the node device 148 to end detection until a new item 110 is placed on the item holder 118.

[0254] At 440, each time the node device 148 finds a defect, the node device 148 transmits defect data to the PLC 146, including the defect type (e.g., void, dirt, etc.) and the defect size.

[0255] If a defect is found, an NG code is sent from the node device 148 to the PLC 146 .

[0256] If no defects are found, an OK code is sent from the node device 148 to the PLC 146. The OK code indicates that no defects were identified in the section of the article 110 being inspected.

[0257] At 442, the PLC 146 determines whether the defect is NG (bad, confirm defect) or OK (reject defect) using the received defect data and defect specifications stored in the PLC 146. The defect specifications include tolerances for a particular section of the article 110 linked to horizontal section values ​​and angle values.

[0258] The PLC 146 makes the decision based on horizontal readings that identify the section, which may be provided from a horizontal movement motor (e.g., stepper motor 1 for moving the imaging unit 122 including the camera 128), and rotational readings from the rotational movement motor of the second article manipulator 120b (stepper motor 2), which identifies the article angle (with reference to the pin / keyway). The PLC 146 determines whether the defect is NG or OK according to the defect specifications for the particular section / angle combination.

[0259] If the PLC determines that the part is NG at 444, the system 100 shuts down and issues an alarm. The shut down and / or alarm is performed according to the shut down and alarm commands generated by the PLC 146. For example, the PLC 146 may generate the alarm commands and send the alarm commands to an alarm system configured to generate or output an alarm upon receiving the alarm commands.

[0260] At 446, if the PLC 146 determines that the item 110 is OK, by receiving an OK code from the node device 148 or by determining that the defects identified by the node device 148 are within an acceptable / tolerance range, inspection continues.

[0261] At 448, the imaging unit 122 and article manipulators 120a, 120b are advanced to the home position in reverse order, which may involve the first actuator (center 1) moving left, the second actuator (center 2) moving right, and the imaging unit 122 moving right and up.

[0262] At 450 , the inspected item 110 is removed from the item holder 118 .

[0263] At 452 , a new item 110 is placed on the item holder 118 .

[0264] In one embodiment, once the PLC 146 determines that the item 110 is NG, the imaging unit manipulator 132 (or robotic arm) moves the camera 128 to an item identifier position where the item identifier on the camshaft is within the imaging zone 124 of the camera 128. An image of the item identifier may be captured by the camera 128. The item identifier is a unique identifier on the item 110 (e.g., etched on the item 110).

[0265] In one embodiment, the item identifier is a 2D data matrix. The 2D data matrix may be similar to a QR code. The item 110 is rotated to make the 2D matrix parallel to the camera 128 so that the detected defects (e.g., defect data) can be associated with the serial number (item identifier) ​​of the item 110. The defects and serial number may be stored in a database. The defects and serial number may be linked within the database. Storage of such data may advantageously provide a record of which particular items were found to be defective and why. Such data may also be used as input in a defect detection analysis process to determine trends or root causes of defects.

[0266] 5, a method 500 for performing in real time the network 156 of node devices 148 of FIG. 1 is shown, according to one embodiment. The method 500 may be implemented by the node devices 148.

[0267] At 508, the method begins.

[0268] At 510, the node device 148 creates and loads an instance of a Basler Camera class (the software API is provided by Basler on their open source github).

[0269] At 512, the selected object detection model is loaded into GPU memory using a specific TensorRT library and / or an optimized openCV CUDA backend.

[0270] At 514, object detection begins.

[0271] 6, there is shown a method 600 of object detection performed by the visual inspection system 100, according to one embodiment. The method 600 may be performed by the node device 148 of FIG.

[0272] The method 600 may be implemented at step 514 of the method 500 of FIG.

[0273] At 610, the method begins.

[0274] At 612 , an image frame is captured from the camera 128 .

[0275] The captured image is pre-processed at 614. To obtain an image compatible with the network 156, pre-processing is performed.

[0276] Image pre-processing may include any one or more of cropping, converting to a NumPy array, changing pixel values ​​to int8 format, changing BGR format to RGB format, and resizing using bilinear interpolation.

[0277] At 616, the pre-processed image is passed through a neural network 156. By passing the image through the network 156, bounding boxes with classes and confidence scores are generated. The bounding boxes enclose objects (defects) located in the image. The classes correspond to defect types.

[0278] At 618, the bounding boxes obtained in 614 are post-processed so that they translate appropriately at the particular image dimensions when the image is upscaled back to its original size after resizing in the pre-processing function.

[0279] At 620, the node device 148 tracks each detected object, including tracking the detected defects across multiple image frames. Tracking the defects in this manner can provide greater certainty that the detected defects are in fact defects and not a one-off inaccurate detection.

[0280] Tracking may allow the algorithm to reduce false positive detections and random one-off detections.

[0281] At 622, while tracking each individual defect, size information of the tracked defect is stored across all frames in which the defect is seen.

[0282] At 624, the node device 148 determines whether the tracked object appears for a minimum number N of consecutive frames without dropping a single frame.

[0283] At 626, a detected object is counted as a true detection if the tracked object is seen for a minimum number N of consecutive frames without dropping a single frame.

[0284] If a detection is deemed a true detection, the average size of the defect is calculated using the size information across all frames the defect appeared in. This technique can reduce the variability due to the bounding box size.

[0285] At 628, the defect data for true detection is sent to the PLC 146. The defect data may include any one or more of defect size, defect location, and defect class. The defect data may be sent from the node to the PLC 146 via TCP / IP over a socket.

[0286] Once the defect data is received by the PLC, the PLC records the stepper motor angle and associates the defect size with the tolerance (in terms of rotation) and the position of the article 110 on that particular section.

[0287] The defect location along with the bounding box coordinates x0, y0, x1, y1 are sent to the PLC 146. The PLC 146 can use the defect location information to identify where the defect was found on that particular section (e.g., a particular lobe or journal of the camshaft being inspected).

[0288] At 630, if the tracked object is not seen for a minimum number N of consecutive frames, the detection is counted as a false positive.

[0289] At 632, the objects that are subject to false positives are ignored (i.e., the defect data is not sent to the PLC).

[0290] The process ends at 634. Steps 612-632 may be repeated until inspection of the article 110 is complete.

[0291] 7-10, exemplary visual inspection images are shown generated by one embodiment of the visual inspection system 100 of FIG 1. An embodiment of the system 100 is configured to image a camshaft and detect paint and void type defects therein.

[0292] The images represent images captured by imaging unit 122, processed by node device 148 using object detection techniques to detect and classify defects, and displayed via user controlled device 150. The images may be stored by node device 148 and / or user controlled device 150 for further review and analysis.

[0293] The images 700 , 800 , 900 , 1000 can be displayed to an operator of the system 100 via the user control device 150 .

[0294] Image 700 of Figure 7 shows a camshaft 710 (i.e., article 110 of Figure 1) inspected by system 100 of Figure 1. In particular, image 700 shows the results of a visual inspection operation performed on section 712 of camshaft 710. Section 712 corresponds to the article section described with reference to article 110 of Figure 1. System 100 is configured to detect paint and void defects.

[0295] System 100 identifies three defects in camshaft 710 based on the captured image data. The defects are enclosed by bounding boxes 714, 716, and 718. The defects include a first paint defect contained within bounding box 714, a second paint defect contained within bounding box 716, and a void defect contained within bounding box 718.

[0296] The bounding boxes 714, 716, 718 are generated by the node device 148 during object detection.

[0297] As shown, the user control device 150 can be configured to associate a unique color indicator with each different defect type. For example, a bounding box surrounding a particular defect type may be imparted with a particular color. This may allow a user to more easily identify and distinguish between different types of defects present within the image 700. In the example image 700, the color green is associated with display information related to a paint defect type and the color red is associated with display information related to a void defect type. Other embodiments may utilize other types of unique indicators to distinguish between defect types.

[0298] Each defect in the image 700 has defect data 720 associated with it.

[0299] The defect data 720 is generated and stored by the node device 148 during the object detection process. The defect data 720 may be passed from the node device 148 to the user control device 150. The user control device 150 is configured to store the defect data 720 and generate a visualization displaying the defect data 720.

[0300] In image 700, user control device 150 displays defect data 720 such that the defect data 720 is linked with bounding boxes of the same defect, making it easy for a user to identify defect data 720 associated with a particular identified defect.

[0301] The defect data 720 includes defect type (or class) 722, defect confidence level 724, defect size 726, and defect location 728. Variations of the system 100 may include more or fewer types of defect data 720. This information may be used by the PLC 146 to understand how each defect compares to a predetermined camshaft defect tolerance.

[0302] Defect type 722 includes the type of defect that was detected. The defect types in image 700 include a paint defect type and a void defect type.

[0303] The defect confidence level 724 represents a confidence level for the detection and classification of the object (i.e., for the assignment of a defect type 722 to the defect).

[0304] The defect size 726 indicates the size of the defect in a particular unit of measure. The defect size 726 in the image 700 is in millimeters.

[0305] Defect location 728 indicates the location of the defect. The defect location includes (x,y) coordinates.

[0306] In the early stages of inspection, when the AI ​​is not yet confident in detecting defects, it may be important to have an operator see the defects in real time as the AI. As the AI ​​inspects more parts, it collects and learns how to identify more defects with greater confidence. The AI's performance will far exceed human capabilities, potentially eliminating the need for human oversight of the AI.

[0307] Image 800 of Figure 8 shows another camshaft 810 that was inspected by the embodiment of system 100 used to generate image 700 of Figure 7. In particular, image 800 shows the results of a visual inspection operation performed on section 812 of camshaft 810.

[0308] Image 800 shows a paint defect and a void defect in section 812. The paint defect is outlined by a first bounding box 814. The void defect is outlined by a second bounding box 818.

[0309] The paint defects and void defects each have defect data 820 associated with them. The defect data 820 is generated by the node device 148 during object detection.

[0310] Image 900 of Figure 9 illustrates yet another camshaft 910 that was inspected by system 100 of Figure 1. In particular, image 900 illustrates the results of a visual inspection operation performed on section 912 of camshaft 910.

[0311] Image 900 shows a paint defect and a void defect in section 912. The paint defect is outlined by a first bounding box 914. The void defect is outlined by a second bounding box 918.

[0312] The paint defects and void defects each have defect data 920 associated with them. The defect data 920 is generated by the node device 148 during object detection.

[0313] Image 1000 of Figure 10 illustrates yet another camshaft 1010 that was inspected by system 100 of Figure 1. In particular, image 1000 illustrates the results of a visual inspection operation performed on section 1012 of camshaft 1010.

[0314] Image 1000 shows a paint defect and a void defect in section 1012. The paint defect is outlined by a first bounding box 1014. The void defect is outlined by a second bounding box 1016.

[0315] The paint defects and void defects each have defect data 1020 associated with them. The defect data 1020 is generated by the node device 148 during object detection.

[0316] 11A-11I, there are shown multiple views of a mechanical inspection subsystem of a visual inspection system of the present disclosure, according to one embodiment. The visual inspection system is adapted to inspect a camshaft. Visual inspection system 100 may be one embodiment of visual inspection system 100 of FIG. 1. The mechanical inspection subsystem may be mechanical inspection subsystem 114 of FIG. 1. The mechanical inspection subsystem of FIGS. 11A-11I may be communicatively connected to a computing system, such as computing system 116 of FIG. 1, configured to perform visual inspection in accordance with the techniques described herein.

[0317] The reference numbers provided in FIGS. 11A-11I correspond to part numbers found in bill of materials 1100j of FIG. 11J.

[0318] FIG. 11A illustrates a first perspective view 1100a of a mechanical testing subsystem according to one embodiment.

[0319] FIG. 11B shows a second perspective view 1100b of the mechanical testing subsystem of FIG. 11A, according to one embodiment.

[0320] FIG. 11C shows a top view 1100c of the mechanical testing subsystem of FIG. 11A, according to one embodiment.

[0321] FIG. 11D illustrates a front view 1100d of the mechanical testing subsystem of FIG. 11A, according to one embodiment.

[0322] FIG. 11E illustrates a rear view 1100e of the mechanical testing subsystem of FIG. 11A, according to one embodiment.

[0323] FIG. 11F illustrates a side view 1100f of the mechanical testing subsystem of FIG. 11A, according to one embodiment.

[0324] 11G illustrates a front view 1100g of an imaging unit of the mechanical inspection subsystem of FIG. 11A, according to one embodiment. The imaging unit can be imaging unit 122 of FIG.

[0325] FIG. 11H shows a side view 1100h of the imaging unit of FIG. 11A, according to one embodiment.

[0326] Figure 11I shows a front view 1100i of the article holder and article manipulator of the mechanical inspection subsystem of Figure 11A, according to one embodiment. The article holder can be article holder 118 of Figure 1. The article manipulator can be article manipulator 120 of Figure 1.

[0327] FIG. 11J illustrates a bill of materials 1100j for the mechanical testing subsystem of FIG. 11A, according to one embodiment.

[0328] 12, there is shown a front view 1200 of a mechanical inspection subsystem of the present disclosure, according to one embodiment. The mechanical inspection subsystem may be the mechanical inspection subsystem 114 of FIG.

[0329] 13, a method 1300 of real-time streaming video analysis is shown, according to one embodiment. The method 100 may be performed by the visual inspection system 100 of FIG.

[0330] At 1308, the method 1300 begins.

[0331] At 1310, the node device 148 connects to the first available camera 128. This may include looking for available USB 3.0 (or other interfaces) and seeing if any cameras are connected.

[0332] The node device 148 can connect to the camera 128 via a camera API. The camera API is a set of specifications that allows application software on the node device 148 to communicate with the camera 128.

[0333] In one embodiment where the camera 128 is a Basler Camera, the node device 148 connects to the first available Basler Camera via the Pylon™ API. The Basler Pylon Camera Software Suite is a collection of drivers and tools for operating Basler cameras.

[0334] At 1312, the node device 148 continuously captures images from the camera 128. The images captured by the node device 148 from the camera 128 are in bitmap format and then converted to NumPy array format using the BGR channel ordering utilized by OpenCV.

[0335] At 1314, the captured image is converted from the camera format to another format suitable for use with OpenCV, etc. In one embodiment, the captured image is converted from Basler Camera format to OpenCV BGR format.

[0336] At 1316, the OpenCV BGR format image data is converted to a NumPy array, which may provide lower memory consumption (NumPy data structures may take up less space) and better runtime behavior.

[0337] At 1318, the NumPy array is sent to a preprocessing function for preprocessing. For example, the NumPy array may be sent to preprocessing function 614 of FIG. 6 for further processing according to method 600 of FIG.

[0338] The pre-processing function pre-processes the image to prepare it for the neural network 156. Pre-processing may include any one or more of cropping, converting to a NumPy array, changing pixel values ​​to int8 format, changing BGR format to RGB format, and resizing using bilinear interpolation.

[0339] 14A-14D, which show a perspective view 1400a, a top view 1400b, a front view 1400c, and a side view 1400d, respectively, of a mechanical inspection subsystem 1414 for use in a visual inspection system, according to one embodiment. The mechanical inspection subsystem 1414 uses robotic automation.

[0340] The mechanical inspection subsystem 1414 may be used as the mechanical inspection subsystem 114 of FIG.

[0341] Elements of Figures 14A-14D may have corresponding elements described in Figure 1. Where an element of mechanical inspection subsystem 114 is referenced by a 1XX reference number and this element has a corresponding element in mechanical inspection subsystem 1414, the corresponding element in mechanical inspection subsystem 1414 will be referenced by a 14XX reference number (i.e., having the same last two digits as the corresponding element in Figure 1). The corresponding elements may perform the same or similar functions.

[0342] The mechanical inspection subsystem 1414 is communicatively coupled to a computing system (e.g., computing system 116 of FIG. 1) to facilitate data transfer between the computing system and the mechanical inspection subsystem 1414 to perform the visual inspection. The computing system 116 performs various control, analysis, and visualization functions as described herein.

[0343] The mechanical testing subsystem 1414 includes a robotic subsystem 1482. The robotic subsystem 1482 includes a robotic arm 1484 and a robotic arm controller (not shown) for controlling the movement of the robotic arm 1484. The robotic arm 1484 is configured to move according to commands from the robotic arm controller. The robotic subsystem 1482 includes a base 1486 for mounting the robotic arm 1484 to a surface. The surface can be on a stationary or movable object.

[0344] The mechanical inspection subsystem 1414 includes an imaging unit 1422 that is attached to a robotic arm 1484. The imaging unit 1422 includes a camera 1428 and an illumination mechanism 1426.

[0345] The robot arm 1484 is configured to move the imaging unit 1422 to capture an image of the inspected article 1410. The robot arm 1484 may be configured to move in three dimensions. During inspection, the article 1410 is engaged by the article manipulators 1420a and 1420b. The article manipulators 1420a, 1420b facilitate rotation of the article 1410 during inspection (e.g., along a line of motion similar to the line of motion 125 of FIG. 1). By rotating the article 1410, the imaging unit 1422 can capture an image of the entire article 1410.

[0346] In a particular case, during an inspection operation, the robot arm 1484 moves the imaging unit 1422 to a section of the article 1410 proximate the article manipulator 1420a and captures multiple images representing a 360° imaging of that section of the article 1410 at that position. The robot arm 1484 then moves the imaging unit 1422 to a second section of the article 1410 for similar 360° imaging at a second position (i.e., the robot arm 1484 moves the imaging unit 1422 further from the article manipulator 1420a to image the second section). This process can be repeated along the length of the article 1410 (e.g., from the end proximate the article manipulator 1420a to the end proximate the article manipulator 1420b).

[0347] 15, there is shown a block diagram 1500 illustrating communication between components in a visual inspection system of the present disclosure, according to one embodiment. In FIG. 15, solid lines between components indicate data communication between the connected components. Such communication may be wired or wireless, depending on the type of connection.

[0348] The visual inspection system includes an AI visual inspection machine 1502 and a production machine 1504. The production machine 1504 may be the last process performed on the article to be inspected. The production machine 1504 generates an output that is provided as an input to the AI ​​visual inspection machine 1502. The AI ​​visual inspection machine 1502 may be the visual inspection system 100 of FIG.

[0349] The production machine 1504 includes a PLC 1506 (designated "PLC2") and an Ethernet / IP module 1508. The Ethernet / IP module 1508 facilitates communication between the PLC 1506 and the components of the AI ​​visual inspection machine 1502. The PLC 1506 is the controller for the production machine 1504. The PLC 1506 communicates with a PLC 1514 (described below) of the inspection machine 1502 for integration purposes (e.g., interfacing I / O signals) that facilitate proper operation of the production machine 1504 and the visual inspection machine 1502.

[0350] The AI ​​visual inspection machine 1502 includes a robot 1510 and an Ethernet / IP module 1512. The robot 1510 may be the robot subsystem 1482 of FIG. 14 and may include a robot arm (e.g., robot arm 1484) for manipulating the imaging unit of the AI ​​visual inspection machine 1502. The Ethernet / IP module 1512 facilitates communication between the robot 1510 and components of the AI ​​visual inspection machine 1502 and the production machine 1504. In particular, the Ethernet / IP module 1512 can communicate with an Ethernet / IP module 1508 to facilitate communication between the robot 1510 and a PLC 1506 of the production machine 1504. In some cases, the PLC 1506 may not need to communicate with the robot 1510. However, in some other cases, Ethernet / IP communication may not be possible directly between the PLC 1506 of the production machine 1504 and the PLC 1514 of the visual inspection machine 1502 (e.g., between a Mitsubishi PLC and a Keyence PLC) and therefore can be achieved through the robot 1510. In such cases, the robot 1510 can act as a medium through which signals can be transferred between the PLC 1506 and the PLC 1514.

[0351] The AI ​​visual inspection machine 1502 also includes a PLC 1514 (denoted as "PLC1"). The PLC 1514 may be the PLC 146 of FIG. 1. The PLC 1514 communicates with other components of the AI ​​visual inspection machine 1502 and the production machine 1504 via an Ethernet / IP module 1516 and an Ethernet module 1518. In particular, the PLC 1514 communicates with the robot 1510 via communication between the Ethernet / IP module 1516 and the Ethernet / IP module 1512. The PLC 1514 may communicate with the PLC 1506 of the production machine 1504 via communication between the Ethernet / IP module 1516 and the Ethernet / IP module 1508.

[0352] The AI ​​visual inspection machine 1502 also includes automation components 1520. The automation components 1520 communicate directly with the PLC 1514. The automation components 1520 include components responsible for automating the inspection process (e.g., components of a mechanical inspection subsystem, such as subsystem 114 of FIG. 1 or 1414 of FIG. 14). The automation components 1520 may include, for example, stepper motors, actuators (e.g., cylinders), sensors, etc. The automation components 1520 communicate with the PLC 1514 via a direct hardwire connection. The automation components 1520 communicate with the PLC 1514 so that the PLC 1514 can execute the desired inspection sequence via I / O signals. For example, the PLC 1514 may initiate an output signal to advance a cylinder (e.g., the first actuator described above). Once the cylinder has advanced, an input is sent back to the PLC 1514 to confirm the "advance" state of the cylinder, etc.

[0353] The AI ​​visual inspection machine 1502 includes an optical component 1522 and an illumination component 1524. The optical component 1522 may be the camera 128 in Fig. 1. The illumination component 1524 may be the illumination mechanism 126 in Fig. 1. The optical component 1522 and the illumination component 1524 constitute an imaging unit of the AI ​​visual inspection machine 1502.

[0354] The lighting components 1524 communicate directly with the PLC 1514 .

[0355] The AI ​​visual inspection machine 1502 includes an AI device 1526. The AI ​​device 1526 may be the node device 148 of FIG. 1. The AI ​​device 1526 communicates directly with the optical component 1522.

[0356] The AI ​​device 1526 communicates with the PLC 1514 via an Ethernet module 1518 .

[0357] The AI ​​visual inspection machine 1502 also includes a display / human machine interface (HMI) 1528 and a human interface 1530. The display / HMI 1528 may be the display 158 or the user control device 150 of Figure 1. The human interface 1530 may be the user interface 160 of Figure 1. The display / HMI 1528 communicates directly with the human interface 1530.

[0358] The display / HMI 1528 communicates with the PLC 1514 via an Ethernet module 1518 .

[0359] The human interface 1530 communicates directly with the AI ​​device 1526.

[0360] Although the above description provides examples of one or more devices, methods, or systems, it will be understood that other devices, methods, or systems may be within the scope of the claims, as interpreted by one of ordinary skill in the art.

Claims

[Claim 1] 1. A system for visual inspection of an article, the system comprising: a camera for acquiring image data of an item to be inspected; a node computing device for receiving the image data from the camera and analyzing the image data using a defect detection model trained to detect at least one defect type, the defect detection model including a machine learning based object detection model configured to receive the image data as an input and generate defect data as an output describing the detected defects; a programmable logic controller ("PLC") device for receiving the defect data from the node computing device and determining whether the defect data is acceptable or unacceptable by comparing the defect data with tolerance data; Including, the system.

Images