Systems and methods for manufacturing quality control using automated visual inspection

Abstract

A system and method for automated visual inspection are provided herein. The method includes providing an inspection image of an article to an object detection model trained to detect at least one defect type in an input image and generate object location data that identifies the location of the detected object in the inspection image; comparing the inspection image and a golden sample image to identify artifacts in the inspection image that correspond to the difference between the inspection image and a golden sample image, defined by artifact location data that describes the location of the artifact in the inspection image; and determining whether the artifact location data matches the object location data according to a predetermined matching criterion.

Description

Technical Field

[0001] The following generally relates generally to manufacturing quality control, and more specifically to systems and methods for automated visual inspection of manufactured articles using artificial intelligence.

Background Art

[0002] Manufacturing quality control is an important part of product manufacturing. Defective products can be costly. Detecting defective products at the appropriate stage of the process can be an important step for businesses to prevent the sale and use of defective articles, determine the root cause associated with defects, and allow such causes to be eliminated.

[0003] One method for guaranteeing manufacturing quality control is through visual inspection of articles. For example, articles can be inspected after manufacturing to identify defects. Defective articles are discarded, and non-defective articles proceed to the next stage of manufacturing. Properly classifying inspected articles as defective or non-defective (i.e., avoiding false positives or false negatives) can be extremely important for controlling costs, increasing efficiency, and limiting liability.

Summary of Invention

Problem to be Solved by Invention

[0004] Visual inspection can be an important part of product manufacturing. Existing visual inspection solutions are difficult to implement in manufacturing and other similar settings when space is limited. Hardware requirements can increase cost and operational complexity.

[0005] Visual inspection techniques include manual and automated visual inspection techniques. Manual visual inspection techniques utilize humans to perform visual inspections, such as physically picking up items and inspecting them using magnification tools. Human visual inspection techniques have limitations in accuracy and speed. Human inspections are prone to overlooking defects. Furthermore, human inspections may include a manual documentation process for visual inspections, which involves documenting identified defects and related information. Such documentation processes can further delay inspections.

[0006] Automated visual inspection technologies, such as those using computer vision and artificial intelligence, offer opportunities to perform visual inspection tasks with increased speed and accuracy. However, such technologies also have limitations. For example, existing automated visual inspection technologies are prone to detecting false positives (e.g., detecting defects that do not exist) and false negatives (e.g., failing to detect defects that do exist). Furthermore, automated visual inspection technologies are typically unsuitable for handling defects on inspected articles that do not reach the level of defect required to classify the article as a defective article, or are not defects at all, and should not trigger a system response as if the article were defective (i.e., abnormal). One example of the latter is when manufactured articles become contaminated during the process, depositing substances on the article surface that can trigger false defect detection.

[0007] Thus, there is a need to provide automated visual inspection solutions that combine the speed and accuracy of conventional automated visual inspection technologies with improved capabilities such as reducing false positives or false negatives, detecting defects and anomalies, verifying automated defect detection (e.g., double-checking results), and identifying new defect types.

[0008] Therefore, there is a need for improved systems and methods for automated visual inspection that overcome at least some of the shortcomings of existing systems and methods. [Means for solving the problem]

[0009] A method for automated visual inspection of articles is provided. The method or a particular step of it may be performed on one or more processors. The method includes providing an inspection image of an article as input to an object detection model, which is trained to detect at least one defect type in an input image and generate object location data that identifies the location of objects detected in an inspection image using the object detection model. The method is defined by artifact location data that describes the location of artifacts in the inspection image and further includes comparing the inspection image with a golden sample image to identify artifacts in the inspection image that correspond to the difference between the inspection image and a golden sample image. The method further includes determining whether the artifact location data matches the object location data according to a predetermined matching criterion.

[0010] The method may further include identifying a detected object as a defect if the artifact location data matches the object location data.

[0011] The method may further include displaying the detected artifacts via a user interface configured to receive input data from the user indicating whether the detected artifact is an exceptional defect or anomaly if the artifact location data does not match the object location data.

[0012] The method may further include receiving input data from the user indicating that the detected artifact is an exceptional defect, and tagging the inspection image as an object detection training sample.

[0013] The method may further include initiating retraining of the object detection model using object detection training samples.

[0014] The golden sample image may also be a reference image representing a clean image of the item.

[0015] The method may further include generating a golden sample image from the inspection image using a generative machine learning model.

[0016] The method may further include: using an object detection model to generate a first defect class label for the detected object; providing at least a portion of the inspection image containing the detected object as input to a classification model; using the classification model to generate a second defect class label for the detected object; and verifying the first defect class label if it matches the second defect class label.

[0017] Comparing the inspection image with the golden sample image may include comparing the inspection image feature map of the inspection image with the golden sample feature map of the golden sample image.

[0018] The method may further include generating an inspection image feature map using a first convolutional neural network ("CNN") and generating a golden sample image feature map using a second CNN.

[0019] A computer system for automated visual inspection of articles is also provided. The system includes an input interface for receiving inspection images of articles, at least one processor, and memory communicating with the processor. At least one processor is configured to provide the inspection image of an article as input to an object detection model, which is trained to detect at least one defect type in the input image and generate object location data that identifies the location of the detected object in the inspection image using the object detection model. At least one processor is further configured to provide the inspection image as input to a first convolutional neural network ("CNN") and use the first CNN to generate an inspection image feature map. At least one processor is further configured to provide a golden sample image of an article as input to a second CNN and use the second CNN to generate a golden sample feature map. At least one processor is further configured to compare the inspection image and the golden sample image to identify artifacts in the inspection image that correspond to the difference between the inspection image and the golden sample image, defined by artifact location data that describes the location of artifacts in the inspection image. At least one processor is further configured to perform a determination, according to a predetermined matching criterion, of whether artifact location data matches object location data.

[0020] At least one processor may be further configured to perform the task of identifying a detected object as a defect if the artifact location data matches the object location data.

[0021] The at least one processor may be further configured to, when the artifact position data does not match the object position data, display the detected artifact via a user interface configured to receive input data from a user indicating whether the detected artifact is an exceptional defect or anomaly.

[0022] The at least one processor may be further configured to receive input data from a user indicating that an undetected feature is a defect, and tag an inspection image as an object detection training sample.

[0023] The at least one processor may be further configured to initiate retraining of an object detection model using the object detection training samples.

[0024] The golden sample image may be a reference image representing a clean image of an article.

[0025] The at least one processor may be further configured to generate a golden sample image from an inspection image using a generative machine learning model.

[0026] The at least one processor may be further configured to: generate a first defect class label for a detected object using an object detection model; provide at least a portion of the inspection image including the detected object as an input to a classification model; generate a second defect class label for the detected object using the classification model; and confirm the first defect class label if the first defect class label matches the second defect class label.

[0027] A method for verifying data output from a machine learning model is also provided. The method is executed on one or more processors. The method includes providing image data as an input to each of a first machine learning model having a first set of model parameters and a second machine learning model having a second set of model parameters different from the first set. The method further comprises detecting, in the image data, a first object comprising first object attribute data that defines an attribute of the first object using the first machine learning model. The method further comprises detecting, in the image data, a second object comprising second object attribute data that defines an attribute of the second object using the second machine learning model. The method further comprises assigning a confirmed status to the first object if the first object attribute data matches the second object attribute data in accordance with a predetermined matching criterion. The method further comprises assigning an unconfirmed status to the first object if the first object attribute data does not match the second object attribute data in accordance with the predetermined matching criterion.

[0028] The attribute may be an object position, and the first object attribute data and the second object attribute data may comprise position data indicating respective positions of the first or second object in the image data.

[0029] The first and second machine learning models may each comprise a neural network.

[0030] The neural network may be a convolutional neural network.

[0031] Other aspects and features will become apparent to those skilled in the art upon reviewing the following description of several exemplary embodiments herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The following drawings contained herein are for illustrating various examples of the articles, methods, and apparatus of the present specification.

[0033] Figure 1 is a schematic diagram of an AI visual inspection system according to one embodiment.

[0034] Figure 2 is a block diagram of a computing device according to one embodiment of the present disclosure.

[0035] Figure 3 is a block diagram of an AI visual inspection system according to one embodiment.

[0036] Figure 4 is a block diagram of an AI visual inspection system having a cluster architecture according to one embodiment.

[0037] Figure 5 is a block diagram of an anomaly detection pipeline for AI visual inspection according to one embodiment.

[0038] Figure 6A is a block diagram of a two-stage object detection model for AI visual inspection according to one embodiment.

[0039] Figure 6B is a block diagram of a two-stage object detection model for AI visual inspection of a camshaft according to one embodiment.

[0040] Figure 7A is a block diagram of a one-stage object detection model for AI visual inspection according to one embodiment.

[0041] Figure 7B is a block diagram of a one-stage object detection model for AI visual inspection of a camshaft according to one embodiment.

[0042] Figure 8 illustrates a comparison of the performance characteristics of two defect classification models according to an embodiment.

[0043] Figure 9A is a graphical representation of a generative model component of a golden sample module using an autoencoder architecture according to one embodiment.

[0044] Figure 9B is a graphical representation of a generative model component of a golden sample module using an autoencoder architecture according to one embodiment.

[0045] Figure 10 is a flowchart of a golden sample pipeline using a generative model according to one embodiment.

[0046] Figure 11 is a graph illustrating unsupervised and semi-supervised learning techniques used by the Golden Sample Module according to one embodiment.

[0047] Figure 12 is an example of an image generated by the visual inspection system of the present disclosure and used by the golden sample component, according to an embodiment.

[0048] Figure 13 shows an example of a golden sample image resulting from the use of an input inspection image and a generated golden sample processing according to one embodiment.

[0049] Figure 14 shows examples of golden sample images obtained by processing a non-generated golden sample and a generated golden sample according to the embodiment.

[0050] Figure 15 is a flowchart of an anomaly detection pipeline for camshaft visual inspection according to one embodiment.

[0051] Figure 16 is a flowchart of an anomaly detection pipeline for camshaft visual inspection according to one embodiment.

[0052] Figure 17 is a graphical representation of the performance characteristics of multiple anomaly detection algorithms according to the embodiment.

[0053] Figure 18 is a block diagram of a computer system for object tracking according to one embodiment.

[0054] Figure 19 is an example of a code representation of the object tracker in Figure 18 according to one embodiment.

[0055] Figure 20 is a flowchart of an object tracking method according to one embodiment.

[0056] Figures 21A to 21D illustrate an example of a user interface for displaying an annotated inspection image processed using object tracking, according to one embodiment.

[0057] Figure 22 is a block diagram of an automated visual inspection pipeline according to one embodiment.

[0058] Figure 23 is a flowchart of an automated visual inspection pipeline for camshaft visual inspection according to one embodiment. [Modes for carrying out the invention]

[0059] Various apparatuses or processes are described below to provide examples of each embodiment of the claims. None of the embodiments described below are limiting to any of the embodiments of the claims, and any of the embodiments of the claims may cover processes or apparatuses different from those described below. The embodiments of the claims are not limited to apparatuses or processes having all the features of any one apparatus or process described below, or to features common to multiple or all of the apparatuses described below.

[0060] Each of the systems described herein may be implemented in a computer program running on a programmable computer having at least one processor, a data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. In non-limiting examples, the programmable computer may be a programmable logic unit, a mainframe computer, a server, and a personal computer, a cloud-based program or system, a laptop, personal data assistance, a mobile phone, a smartphone, or a tablet device.

[0061] Each program is preferably implemented in a high-level procedural or object-oriented programming and / or scripting language for communicating with a computer system. However, the program may be implemented in assembly or machine language as needed. The language may be a compiled or interpreted language. Each such computer program is preferably stored on a storage medium or a device readable by a general-purpose or dedicated programmable computer, and the computer is configured and operated when the storage medium or device is read by the computer in order to execute the procedures described herein.

[0062] The description of one embodiment having several components communicating with each other does not mean that all such components are required. On the contrary, various optional components are described to illustrate the broad scope of possible embodiments of the present invention.

[0063] Furthermore, while the steps of a process, the steps of a method, the algorithm, etc., may be described in order (in the disclosure and / or claims), such processes, methods, and algorithms may be configured to work in different orders. In other words, no sequence or order of steps that may be described necessarily requires that the steps be performed in that order. The steps of the process described herein may be performed in any practical order. Furthermore, some steps may be performed simultaneously.

[0064] Where a single device or article is described herein, it is clear that multiple devices / articles (whether they work together or not) may be used in place of the single device / article. Similarly, where multiple devices or articles are described herein (whether they work together or not), it is clear that a single device / article may be used in place of multiple devices or articles.

[0065] Referring to Figure 1, a block diagram showing a visual inspection system 10 according to one embodiment is shown. The system 10 includes an AI visual inspection device 12 that communicates with a camera device 14, an operator device 16, a programmable logic controller ("PLC") device 18, and an analysis server 22 via a network 20. The system 10 also includes an object detection database server 24 that can access and / or store object detection training data for training the AI ​​visual inspection device 12. The system 10 also includes an analysis client device 26 that communicates with the analysis server 22, which may be part of the operator device 16 or separate from the operator device 16. Devices 24 and 26 may be connected via the network 20 in a communicative manner to one or more other devices 12, 14, 16, 18, 22, 24, and 26.

[0066] Devices 12, 14, 16, 18, 22, 24, and 26 may be server computers, node computing devices (e.g., JETSON computing devices), embedded devices, desktop computers, notebook computers, tablets, PDAs, smartphones, or other computing devices. Devices 12, 14, 16, 18, 22, 24, and 26 may include connections to a network 20, such as a wired or wireless connection to the Internet. In some cases, the network 20 may include other types of computers or communication networks. Devices 12, 14, 16, 18, 22, 24, and 26 may include one or more memory, sub-storage devices, processors, input devices, display devices, and output devices. The memory may include random access memory (RAM) or a similar type of memory. The memory may also store one or more applications for execution by the processor. The applications may correspond to software modules having computer executable instructions for performing processing for the functions described below. The secondary storage device may include a hard disk drive, floppy disk drive, CD drive, DVD drive, Blu-ray drive, or other types of non-volatile data storage. The processor may execute applications, computer-readable instructions, or programs. Applications, computer-readable instructions, or programs may be stored in memory or secondary storage, or received from the Internet or other networks.

[0067] The input devices may include any devices for inputting information into devices 12, 14, 16, 18, 22, 24, and 26. For example, an input device may be a keyboard, keypad, cursor control device, touchscreen, camera, or microphone. The display devices may include any type of device for presenting visual information. For example, a display device may be a computer monitor, flat-screen display, projector, or display panel. The output devices may include any type of device for presenting a hard copy of information (e.g., a printer). The output devices may also include other types of output devices such as speakers. In some cases, devices 12, 14, 16, 18, 22, 24, and 26 may include any multiple of the following: processors, applications, software modules, second storage devices, network connections, input devices, output devices, and display devices.

[0068] Devices 12, 14, 16, 18, 22, 24, and 26 are described with various components, but a person skilled in the art will understand that in some cases devices 12, 14, 16, 18, 22, 24, and 26 may include fewer, additional, or different components. In addition, aspects of the implementation of devices 12, 14, 16, 18, 22, 24, and 26 may be described as being stored in memory, but a person skilled in the art will understand that these aspects may be stored on and / or read from computer-readable media such as other types of computer program products or secondary storage devices (including hard disks, floppy disks, CDs or DVDs, carriers from the Internet or other networks, or other forms of RAM or ROM). Computer-readable media may include a processor for executing instructions and / or specific methods for controlling devices 12, 14, 16, 18, 22, 24, and 26.

[0069] Devices 12, 14, 16, 18, 22, 24, and 26 may be described as performing specific actions. It is understood that at least one of these devices may perform an action automatically or in response to interaction by the user of that device. That is, the user of a device may operate one or more input devices (e.g., a touchscreen, mouse, or buttons) to cause the device to perform the action described. In many cases, this aspect is not described below but is implicitly understood.

[0070] For example, devices 12, 14, 16, 18, 22, 24, and 26 may send information to one or more other devices 12, 14, 16, 18, 22, 24, and 26, as described below. For example, a user using operator device 16 may interact with a user interface displayed on the display of device 16 by manipulating one or more inputs (e.g., a mouse and keyboard). Generally, a device may receive the user interface from the network 20 (e.g., in the form of a web page). Alternatively, or in addition, the user interface may be stored locally on the device (e.g., a web page cache or a mobile application).

[0071] Devices 12, 14, 16, 18, 22, 24, and 26 may be configured to receive multiple pieces of information from at least one of the multiple devices 12, 14, 16, 18, 22, 24, and 26.

[0072] Each device 12, 14, 16, 18, 22, 24, 26 that receives information may store that information in a storage database. The storage may correspond to the secondary storage of one or more other devices 12, 14, 16, 18, 22, 24, 26. Generally, the storage database may be any suitable storage device such as a hard disk drive, solid state drive, memory card, or disk (e.g., CD, DVD, or Blu-ray). The storage database may also be locally connected to devices 12, 14, 16, 18, 22, 24, 26. In some cases, the storage database may be located away from devices 12, 14, 16, 18, 22, 24, 26 and accessible via a network or the like. In some cases, the storage database may comprise one or more storage devices located at a network cloud storage provider.

[0073] The AI ​​visual inspection device 12 may also be a dedicated device specifically designed to perform object (e.g., defect) detection tasks, object (e.g., defect) classification tasks, golden sample analysis tasks, object (e.g., defect) tracking tasks, and other related data processing tasks using inspection images acquired by the camera device 14.

[0074] The analysis server 24 may also be a dedicated device specifically designed to receive, process, store, and analyze output data from object detection tasks, defect classification tasks, golden sample analysis tasks, object tracking, or other related data processing tasks performed by the AI ​​visual inspection device 12. The analysis server 24 may include, or communicate with, an analysis database that stores various data received from at least one of the devices 12, 14, 16, 18, 22, 24, 26, such as the AI ​​visual inspection device 12, or data generated by the analysis server 24 using data received from one or more devices.

[0075] Referring to Figure 2, a simplified block diagram of the components of device 100, such as a mobile device or portable electronic device, is shown. Device 100 may be, for example, any of devices 12, 14, 16, 18, 22, 24, or 26 in Figure 1. Device 100 includes several components, such as a processor 102 that controls the operation of device 100. Communication functions, including data communication, voice communication, or both, may be implemented through a communication subsystem 104. Data received by device 100 may be decompressed and decoded by a decoder 106. The communication subsystem 104 may receive messages from or send messages to the wireless network 150.

[0076] The wireless network 150 may include, but is not limited to, any type of wireless network, including, a data-centric wireless network, a voice-centric wireless network, and a dual-mode network that supports both voice and data communications.

[0077] Device 100 may be a battery-powered device and may include a battery interface 142 for receiving one or more rechargeable batteries 144, as shown in the figure.

[0078] The processor 102 also interacts with additional subsystems such as random access memory (RAM) 108, flash memory 110, display 112 (for example, having a contact-sensing overlay 114 connected to an electronic controller 116, together with a contact-sensing display 118), actuator assembly 120, one or more optional force sensors 122, auxiliary input / output (I / O) subsystem 124, data port 126, speaker 128, microphone 130, short-range communication system 132, and other device subsystems 134.

[0079] In some embodiments, user interaction with a graphical user interface may be performed through a contact-sensing overlay 114. The processor 102 may interact with the contact-sensing overlay 114 via an electronic controller 116. Information such as text, characters, symbols, images, icons, and other items generated by the processor 102 and displayed or depicted on a portable electronic device may be displayed on the contact-sensing display 118.

[0080] The processor 102 may interact with the accelerometer 136, as shown in Figure 1. The accelerometer 136 may be used to detect the direction of gravity or the reaction force originating from gravity.

[0081] In order to identify a network access subscriber according to this embodiment, device 100 may use a subscriber identification module or removable user identification module (SIM / RUIM) card 138 inserted into the SIM / RUIM interface 140 for communication with the network (e.g., wireless network 150). Alternatively, user identification information may be programmed into flash memory 110 or implemented using other techniques.

[0082] Device 100 also includes an operating system 146 and software components 148 executed by a processor 102, which may be stored in a persistent data storage device such as flash memory 110. Additional applications may be loaded onto device 100 through a wireless network 150, a spare I / O subsystem 124, a data port 126, a short-range communication subsystem 132, or any other suitable device subsystem 134.

[0083] During use, received signals such as text messages, email messages, web page downloads, or other data may be processed by the communication subsystem 104 and input to the processor 102. The processor 102 then processes the received signals for output to the display 112 or the auxiliary I / O subsystem 124. Subscribers may create data items, such as email messages, which may be transmitted over the wireless network 150 via the communication subsystem 104.

[0084] For voice communication, the overall operation of the portable electronic device 100 may be the same. The speaker 128 may output audio information converted from an electrical signal, and the microphone 130 may convert the audio information into an electrical signal for processing.

[0085] Referring to Figure 3, an automated visual inspection system 300 according to one embodiment is shown. The system 300 can be used to visually inspect target articles. Visual inspection may include defect detection and classification and anomaly detection. Figure 3 shows an example architecture of hardware components for the visual inspection system 300.

[0086] System 300 includes a camera 304. The camera 304 acquires image data of a target item 306. The image data may include a single image or multiple images. Multiple images (frames) may be acquired as video by the camera 304. The camera 304 and the target item 306 may move relative to each other to image an area of ​​the target item 306. For example, to provide proper inspection from multiple angles, the target item 306 may be rotated so that multiple images are acquired by the camera 304 at different positions on the target item 306. The camera 304 may be configured to acquire multiple frames taken at each target item position (e.g., if the target item 306 rotates relative to the camera 304). The camera 304 may be a USB 3.0 camera or an Internet Protocol ("IP") camera.

[0087] The target article 306 may be any physical article on which the user of the system 300 wishes to perform a visual inspection. The target article 306 may be a manufactured or machine-made article. The target article 306 may have a curved outer surface. The target article 306 may have a shape that requires it to be rotated relative to the camera 304 so that the surface of the target article 306 is inspected. In certain cases, the target article 306 may be a camshaft.

[0088] The target article 306 may have various physical properties and characteristics that pose a challenge when performing the visual inspection task and are configured to be addressed by the system 300. For example, the target article 306 may have different surface finishes. The target article 306 may cause refraction of illumination (e.g., under inspection conditions). The target article 306 may have variations in defect tolerance depending on the position on the target article 306. The target article 306 may be a complex article (e.g., having a complex shape or structure) which may affect the illumination and cycle time for visual inspection. The target article 306 may be prone to variations in defect characteristics. The target article 306 may have asymmetric characteristics. Asymmetric characteristics that may affect the focal length with respect to the camera position may require advanced robotic manipulation to obtain a proper image of the target article 306.

[0089] The target article 306 may be a cylindrical metal object of variable length. The target article 306 may have a machined surface with specific tolerances for the texture and size of defects. The target article 306 may be an article that can be cleaned or covered (fully or partially) with a transparent or translucent material (liquid) during visual inspection. The appearance of the defects may be predetermined or collected during the operation of system 300 for further training.

[0090] The target article 306 may be susceptible to defects that develop during the manufacturing or machining process. These defects may be characterized as an unacceptable deviation from a “perfect” or “good” article. A target article 306 having defects is interpreted as defective, unacceptable, or “bad” (“NG”).

[0091] The target item 306 may be susceptible to anomalies that develop during the manufacturing and production process. The anomalies may be characterized as deviations from an acceptable good product. In other words, the anomalies are deviations that are not significant enough to classify the part as defective and should not trigger handling as a defective target item.

[0092] The system 300 inspects article 306 and determines whether article 306 is defective or not. Article 306 may be classified by the system 300 as defective or not defective.

[0093] By identifying whether article 306 is defective or not, the inspected articles may be treated differently based on the results of the visual inspection. Defective articles 306 may be discarded or removed from further processing. Non-defective articles 306 may proceed to further processing.

[0094] In general, Article 306 is an article in which defects are undesirable. Defects in Article 306 can lead to a decrease in the functional performance of Article 306 or a larger article (e.g., a system or device) that contains Article 306 as a component. Defects in Article 306 can reduce the visual appeal of the article. Detecting defective products can be a critical step for businesses to prevent the sale and use of defective articles and to determine the root cause of the defect and eliminate such cause.

[0095] Article 306 may be an assembled article. Article 306 may be a manufactured article that is prone to defects that develop during the manufacturing process. Article 306 may be an article that derives some value from its appearance, and in which certain defects may adversely affect its appearance. Defects in Article 306 may develop during the manufacture of Article 306 itself or during some other process (e.g., transport, testing).

[0096] Article 306 may be made of one or more materials such as metal, iron, plastic, composite, wood, glass, etc.

[0097] Article 306 may be uniform or non-uniform in size and shape. Article 306 may have a curved outer surface.

[0098] Article 306 may include multiple sections. Article sections may be further divided into article subsections. Article sections (or subsections) may be determined based on the appearance or function of the article. Article sections may be determined to allow for better visual inspection of Article 306 and to better identify unacceptable defective articles.

[0099] Article sections may correspond to different parts of article 306 having different functions. Different sections may have similar or different dimensions. In some cases, article 306 may include several different section types, each appearing once or more times in article 306. Sections may have regular or irregular shapes. Different sections may have different defect specifications (i.e., tolerances for specific defects).

[0100] Article 306 may be a standalone article intended for use on its own, or it may be a component of a larger article (e.g., a component of a larger device).

[0101] Article 306 may be prone to multiple types or classes of defects detectable using System 300. Examples of defect types may include paint, porosity, dents, scratches, sludge, etc. The defect types may vary depending on Article 306. For example, the defect types may be specific to Article 306 based on the manufacturing process or material composition of Article 306. Defects in Article 306 may be acquired during its own manufacturing or through subsequent processing of Article 306.

[0102] Article 306 may also be powertrain components for automobiles and / or motorcycles. Powertrain components may be camshafts, differential assemblies, transmission gears, linkages, suspension components, or any of these parts or components.

[0103] In a particular embodiment of system 300, article 306 is a camshaft. Generally, a camshaft is a shaft on which a cam is mounted, or a shaft integrally configured with a cam. Camshafts can be used as mechanical components of an internal combustion engine. Camshafts open and close the intake and exhaust valves of the engine at appropriate times in a precisely defined sequence.

[0104] A camshaft may include multiple different components or parts, each providing a specific function and having specific dimensions. Camshaft components may include at least one of journals, lobes, ends, and bearings.

[0105] The camshaft may be divided into sections for inspection using System 300. The sections may correspond to different components of the camshaft (e.g., lobe section, journal section). The camshaft may have between 10 and 17 sections for inspection. The sections between 10 and 17 may include lobes and journals.

[0106] The system 300 may be configured to acquire 1300 range images per camshaft. The system 300 may acquire 50 range images per section (for example, a section between 13 and 17 per camshaft). For each section, an image may be acquired every 7.2 degrees of rotation of the camshaft. This allows for a 360-degree rotation every 2 seconds, while giving the AI ​​process sufficient degrees of freedom to properly track defects across the 50 images in each section.

[0107] The camshaft may include a cylindrical rod extending over the length of the cylinder bank, having multiple rectangular lobes protruding from the rod. The number of lobes may correspond to the number of valves (e.g., one lobe for each valve). The main journal holds the rotating camshaft in place in the engine bay.

[0108] The camshaft may be constructed from any suitable camshaft material. The camshaft may be made from chilled cast iron or billet steel.

[0109] Camshafts may be prone to defects that develop during the manufacturing process of the camshaft or its components, or through other processes. Defect types may include paint, porosity, sludge, etc. Different types of defects may develop in different components or sections of the camshaft. Defects may affect different components in different ways. This is because, for example, different components may have different tolerances for defects because they perform different functions. For example, defects in lobes and journals may be treated in different ways and have different tolerances. In some cases, subsections of a component may have different specifications. For example, lobes (which are egg-shaped) may have different defect specifications based on the location of defects on the lobes.

[0110] In some cases, the camshaft may have a porous structure with a small porosity of about 0.4 mm. The porosity may be position-dependent. An example is shown in the table below. [Table 1]

[0111] The camshaft section may have a non-uniform shape. This can result in irregular light reflection. In some cases, the camshaft may be covered with a rust inhibitor (clear). Sludge or dark oil drops may occur randomly on the camshaft. In some cases, the sludge may cover porosity. The defects (e.g., porosity, sludge) may have different sizes and shapes.

[0112] Camera 304 is connected to worker node device 310 via communication link 312 in a communicative manner. Worker node device 310 may be the AI ​​visual inspection device 12 shown in Figure 1.

[0113] Camera 304 sends image data to worker node device 310 via communication link 312. In one embodiment, camera 304 acquires an image frame at the current target item position and sends the image frame to worker node device 310.

[0114] The worker node device 310 includes an object detection component 314. The object detection component 314 is configured to perform an object detection process on image data to detect defects (objects) in the image data. Generally, the object detection process determines whether one or more defects are present in the image data.

[0115] The object detection component 314 is further configured to perform a defect classification process for classifying defect types (i.e., assigning defect classes to detected defects). Defect classification may be performed on input provided by the object detection process. The defect classification process is triggered in response to the object detection process detecting one or more defects in the image data. The defect classification process assigns class labels (e.g., defect names or defect types such as "scratch") to the defects provided by the object detection process. In some cases, the object detection component 314 includes an object detection model and an object classification model. The object detection model generates an object class for each detected object. Image data containing the detected objects is provided to the classification model, which outputs object classes. The object detection component 314 compares the object class determined by the object detection model with the object class determined by the classification model to verify the class label for the object. If the class label is not verified by the classification model, the object detection component 314 may be configured to ignore the detection.

[0116] The worker node device 310 also includes a golden sample component 316. The golden sample component 316 is configured to perform a golden sample analysis process on the inspection image received from the camera 304. Thus, the worker node device 310 may be configured to pass the inspection image to the golden sample component 316.

[0117] The golden sample analysis process includes analyzing the inspection image by referencing the golden sample image to determine the difference. The difference between the inspection image and the golden sample image is identified by the golden sample component 316 and can be used in subsequent processing operations such as detecting anomalies and identifying new types or classes of defects. For example, the output of the golden sample component 316 (identified difference or object) and the output of the object detection component 314 (detected object) may be compared.

[0118] The golden sample image may be generated by the golden sample component 316. In one embodiment, the golden sample image is generated from the inspection image.

[0119] The object detection component 314 and the golden sample component 316 may be connected in a communicative manner via a communication link 318. The communication link 318 may include an application programming interface (API), etc.

[0120] The object detection component 314 and the golden sample component 316 may each have their own dedicated hardware components. For example, the object detection component 314 may run on a first embedded device, and the golden sample component 316 may run on a second embedded device. The embedded device may be an embedded device specifically configured to perform artificial intelligence type tasks. In one embodiment, the embedded device is a JETSON box.

[0121] System 300 also includes a PLC device 320. The PLC device 320 is connected to the worker node device 310 via a communication link 322.

[0122] The PLC device 320 is configured to control the operation and physical processing of the target article 306. This may be done by sending and receiving control commands to and from an article handling unit (not shown) via a communication link 321. Such operation and physical processing may include rotating or otherwise driving the target article 306 for the placement and removal of the target article 306 in an imaging or inspection area. An example of a command sent by the PLC 320 to the article handling unit via the communication link 321 may be "rotate the target article by "n" degrees". In some cases, the transmission of such commands may depend on information received from a worker node device 310 (e.g., an object detection component 314).

[0123] The PLC320 may store defect tolerance data. The defect tolerance data may include a defect class identifier specific to a particular defect class and one or more tolerance values ​​linked to the defect class identifier. In other embodiments, the defect tolerance data may be stored on another device, such as a worker node device 310. The defect tolerance data may be stored in a defect tolerance database. The defect tolerance data in the defect tolerance database may be referenced using a defect class identifier to facilitate the retrieval of tolerance data values ​​for comparison with data generated by the worker node device 310 (e.g., via components 314, 316).

[0124] For example, in one embodiment, the PLC320 is configured to receive data indicating the results of the defect detection process from the worker node device 310 via the communication link 322. For example, if a defect is detected by the object detection component 314, the defect data may be sent to the PLC320. The PLC320 stores the defect tolerance data. The PLC320 analyzes the defect data against the tolerance data and determines whether the target article 306 is defective (e.g., "NG") or within the tolerance range (e.g., "OK"). The PLC320 may also send a signal to the worker node device 310 indicating the result of the tolerance analysis. If the PLC320 determines that the defect data is outside the tolerance range, the PLC320 may stop the inspection of the target article 306 and start processing for removing the defective target article and installing a new target article. The PLC320 may generate a control signal to stop the inspection of the target article and may send a control signal to the actuator responsible for operating the target article 306.

[0125] If the object detection component 314 does not detect a defect in the inspection image, the worker node device 310 sends a signal to the PLC 320 (for example, via the object detection component 314) indicating the result of the object detection process, that no defects were found in the image (i.e., "OK"). Upon receiving the "OK" message, the PLC 320 sends a control signal to the actuator or manipulator of the target article 306 to adjust the current inspection position of the target article 306 (for example, to rotate the target article 306 by "X" degrees).

[0126] In other embodiments, defect tolerance data may be stored in the worker node device 310, and tolerance analysis may be performed by the worker node device 310. The worker node device 310 may also send a signal to the PLC 320 indicating whether or not the target item is defective.

[0127] System 300 also includes an operator device 324. The operator device 324 is communicated with the worker node device 310 via a communication link 326.

[0128] The operator device 324 includes a user interface component (or module) (e.g., a human / machine interface). The operator device 324 receives data from the worker node device 310 via a communication link 326. The received data may include output data from the object detection component 314 and the golden sample component 316 of the worker node device 310. For example, the output data may include an annotated inspection image containing artifact data. The artifact data may include location information (e.g., coordinates, bounding boxes) and label information so that artifacts (e.g., defects, anomalies) in the inspection image identified by the worker node device 310 can be visually identified in the displayed image.

[0129] The worker node device 310 or operator device 324 may include automated image annotation software for automatically assigning metadata to digital inspection images, which includes data generated by components 314, 316.

[0130] The operator device 324 provides output data from the worker node device 310 to a user interface component that generates a user interface screen for displaying the annotated inspection image. For example, the inspection image may be annotated with metadata including defect data generated by a component such as defect location information (e.g., bounding box coordinates, centroid coordinates), defect size data, and defect class information.

[0131] The user interface component of the operator device 324 may depict one or more user interface elements for receiving input from the operator. For example, the user interface component may provide a "yes / no" or similar binary option for receiving user input data indicating an option selection. In certain cases, the user interface may present and highlight a specific object detected by the worker node device 310 in the annotated inspection image, and may ask whether the object is abnormal or not (and may receive corresponding input from the user).

[0132] Depending on the input data received from the user, the annotated inspection image (or a portion thereof) may be routed differently within the system 300. For example, if the user interface component of the operator device 324 receives specific input data (e.g., a "no" response to a question about whether a given artifact is abnormal, such as clicking a user interface element labeled "no"), the operator device 324 or the worker node device 310 may be configured to send the annotated inspection image (or a subset of the image data) to the object detection training database 330 via communication links 332 or 333, respectively. The data received by the object detection training database 330 may be recorded as training samples for further training datasets that can be used to further train one or more artificial intelligence components of the worker node device 310.

[0133] System 300 also includes a server node device 334. The server node device 334 is connected to the worker node device 310 via a communication link 336. In particular, the server node device communicates with the object detection component 314 of the worker node device 310 via the communication link 336. The server node device 334 may include a Jetson device or the like.

[0134] The server node device 334 receives visual inspection data from the worker node device 310. The visual inspection data includes output data (or "defect data") from the object detection component 314. The defect data may include whether or not a defect was found, a unique defect identifier for each detected defect, the number of defects found, the presence or absence of defects in the target item, the location of the defects (defect location data such as bounding box coordinates), a defect class identifier, etc. The server node device 334 includes a visual inspection analysis component configured to analyze the received defect data.

[0135] The server node device 334 is connected to the client device 338 via a communication link 340. In some cases, the client device 338 may include the server node device 334 (i.e., the server node device is a component of the client device).

[0136] The server node device 334 is also connected to the analysis database 342 via a communication link 344. The analysis database 342 stores analysis data and visual inspection output data (e.g., defect data) from the worker node device 310.

[0137] Defect data may be stored so that a database record is generated and maintained for each inspected target item 306. The record may include a target item identifier (e.g., a code on the item acquired by a camera), which may be obtained from the target item itself, or it may be automatically generated by the server node device 334. Various defect data may be associated with or linked to the database record for the target item 306. Each defect may be assigned a unique identifier to which other data about the defect may be linked.

[0138] Analysis data may be generated from visual inspection data by the server node device 334. Analysis data may also be generated by applying statistical analysis techniques to the visual inspection data. Analysis data may provide operators or other users with insights into the decisions made by the system 300 across a large number of target items 306.

[0139] The client device 338 includes a user interface component configured to provide a graphical user interface via a display 346. The user interface component receives analysis data from the server node device 334 and displays the analysis data via the graphical user interface on the display 346. In some cases, when the server node device 334 receives visual inspection data from the worker node device 310, the server node device 334 and the user interface component are configured to update the graphical user interface in real time.

[0140] Referring to Figure 4, a visual inspection system 400 according to another embodiment is shown. The visual inspection system 400 has a cluster architecture. If the components of system 400 are assigned reference digits in the 300 series, such components function similarly to components with similar digits, as described with reference to Figure 3.

[0141] The visual inspection system 400 includes a cluster 402. The cluster 402 includes a plurality of cluster units 404. Each cluster unit 404 includes a camera 304 and a worker node device 310. The camera 304 and the worker node device 310 are connected communicably via a communication link 312, as shown in the system 300 in Figure 3.

[0142] Each worker node device 310 in cluster 402 is connected to the same PLC device 320 via a communication link 322, enabling communication between them.

[0143] In one embodiment, the system 400 is configured such that there is one PLC device 320 communicating with worker node devices 310 between 2 and 16.

[0144] System 400 includes a server node device 334. Each worker node device 310 is connected to the server node device 334 to provide real-time streaming data.

[0145] The server node device 334 is connected to a client device 338, including a display 346, in a communicative manner.

[0146] The cluster architecture of System 400 can offer certain advantages. The cluster architecture allows for the management of a set of devices 310. If one edge device (i.e., a worker node device 310) fails, all parts are replaceable, allowing the user to operate with minimal or zero downtime. In addition, the cluster architecture facilitates the management of groups of devices in a cluster, simplifying the scaling-up of the visual inspection system.

[0147] Referring to Figure 5, a visual inspection pipeline 500 according to one embodiment is shown. The pipeline 500 is implemented by one or more computing devices, such as the worker node device 310 in Figure 3. The pipeline 500 performs defect detection and classification and anomaly detection.

[0148] The pipeline 500 includes an inspection image 502. The inspection image 502 comprises image data of a target article, such as the target article 306 in Figure 3, acquired by a camera device (e.g., camera device 304 in Figure 3). The inspection image 502 may be a single frame acquired by the camera device 304. In general, the pipeline 500 may be configured to receive multiple inspection images 502 for a given target article 306. However, in some embodiments, a single inspection image 502 is sufficient.

[0149] The inspection image 502 may be received from the camera device 304 via the communication link 312, or it may be received in real time each time an image is acquired.

[0150] The inspection image 502 is provided to the object detection module 504. The object detection module 504 includes an object detection model for performing an object detection task. The object detection model performs computer vision and image processing to detect instances of semantic objects of a specific class in the digital image (inspection image). Generally, the object detection module 504 takes the inspection image as input, passes the inspection image through the object detection model (e.g., a CNN), and generates object detection (OD) output data. The OD output data may also be in the form of an annotated inspection image containing metadata describing the objects detected in the inspection image 502.

[0151] An object detection model is trained and configured to find defects in the inspection image 502. The object detection model may function as a classifier, having one class for each defect type. Images 502 containing one or more defects may be labeled "NG". Images in which no defects were detected may be labeled "OK".

[0152] The object detection model may be configured to output the object class, object location (e.g., bounding box coordinates), and confidence score for each detected object. For example, for each detected object, the object detection model may return an array of four numbers (e.g., [Top, Left, Bottom, Right]) representing the bounding box (location data) surrounding the location of the detected object. The Top value represents the distance in pixels from the top of the image to the top edge of the rectangle. The Left value represents the distance from the left side of the input image to the left edge. The other values ​​(Bottom and Right) represent the bottom and right edges in a similar manner.

[0153] The inspection image 502, labeled "OK," is not passed to the defect classifier (e.g., defect classification module 520).

[0154] The inspection image 502 labeled "NG" contains one or more defects identified by the object detection model and is passed to the defect classifier (e.g., defect classification module 520).

[0155] The OD output data 518 for the "NG" inspection image may include inspection image data and defect data. The defect data may be stored as annotation data or metadata for the inspection image 502.

[0156] The defect data may include a list of detected defects. The list of detected defects may include data about each defect detected by the object detector model (e.g., class label, bounding box coordinates, and other positional data). The defect data may include a list of one or more bounding boxes corresponding to one or more defects detected by the object detector model. Each bounding box represents an area of ​​the inspection image defined by x and y coordinates surrounding the defect detected by the object detector model. In one embodiment, the defect data includes defect positional data (e.g., bounding box coordinates) and a defect class label for each detected defect.

[0157] The defect data may include the defect count for inspection image 502 (i.e., the number of defects detected).

[0158] The defect data may include a defect identifier assigned to the detected defect. It should be noted that in some embodiments, defects detected by the object detection model are "candidate defects." This is the case, for example, when the defect classifier includes an "OK" class representing defects detected by the object detector model that are not assigned to a defect class. In other words, defects detected by the object detection module 504 may later be determined and identified as not being defects by other components of the system (e.g., defect classifier 520, anomaly detector module 540).

[0159] If the object detection module 504 does not detect any defects in the inspection image 502, the object detection module 504 may generate an "OK" signal or message for the inspection image 502. The "OK" signal may be sent to one or more components of the system, such as the PLC 320.

[0160] In one embodiment, the object detection module 504 can perform object detection on an "800pi x 600pi" inspection image in 40ms.

[0161] In Figure 5, the object detection model is a convolutional neural network (CNN) 505. In other embodiments, the object detection model may be any suitable object detector that implements machine learning, deep learning, or computer vision techniques.

[0162] CNN505 includes multiple components or layers. The CNN components include an object proposal component 506, a feature map component 508, a region of interest (ROI) pooling component 510, a fully connected layer component 512, a softmax component 514, and a regressor component 516. The softmax component 514 provides a class label output for each object detected in image 502. The regressor component 516 provides final bounding box coordinates for each object detected in image 502. The bounding box coordinates define the target location (i.e., the detected object). The bounding box is a rectangular box that may be determined by the x and y axis coordinates at the top-left corner of the rectangle and the x and y axis coordinates at the bottom-right corner. The class label and final bounding box coordinate outputs may be combined into an object detection output by the object detection module 504. The object detection output may include a confidence score (e.g., between "0" and "1").

[0163] The object detection module 504 may be further configured to crop or isolate a subset of the inspection image 502 in order to isolate a region of interest in the inspection image 502 containing the detected defect. The object detection module 504 may use an image cropping function or the like to generate a subset of the inspection image data corresponding to the defect region. The cropping function may be configured to receive a bounding box and the inspection image 502 from the object detection module 504 and crop the portion of the inspection image 502 (a subset of the image data) corresponding to the bounding box. As a result, the cropping function generates a defect image 524 which is a subset or part of the inspection image. The defect image 524 may be generated at a different pixel resolution than the pixel resolution of the inspection image input to the object detection module 504, or it may be converted to that pixel resolution. In one embodiment, the defect image 524 is within the range of "200pi x 200pi". The object detection module 504 may generate a "200pi x 200pi" defect image from an "800pi x 600pi" inspection image 502.

[0164] If multiple defects are detected in the inspection image 502, the object detection module 504 performs a cropping function to generate a defect image 502 (for example, a region of interest defined by a bounding box) for each detected defect.

[0165] The annotated inspection image 518 generated by the object detection module 504 is passed to the defect classification module 520. This may include generating one or more defect images 524 from the annotated inspection image 518 and passing the defect images 524 to the defect classification module 520.

[0166] The defect classification module includes a defect classification model 522. The defect classification model 522 may be a neural network. In pipeline 500, the classification model 522 is a convolutional neural network. Other classification models using neural networks or other machine or deep learning techniques may be used. The defect classification model 522 may be trained using training data containing labeled defect images.

[0167] The defect classification module 520 is configured to analyze a defect image 524 using a trained defect classification model 522 and generate a class label or assignment for the defect image 524. The defect image 524 may also be a cropped defect image.

[0168] The defect classification model 522 includes an output layer with multiple output nodes. Each output node in the output layer corresponds to a defect class associated with a defect type. For example, in the camshaft visual inspection example, the defect classes may include at least one of porosity, dents, scratches, entrapment scale, patches, crazing, uneven surfaces, and inclusions. In some cases, such as when different subtypes of defects are possible and it is not useful or efficient to generalize the defect classifier to classify subtypes under the same class, a particular defect type may have multiple classes (i.e., subclasses of a larger class). One such example is when there are different classes for different types of porous defects. The classification model 522 also includes output nodes corresponding to the "no defect" or "OK" class.

[0169] The defect classification model 522 may be configured to generate a probability distribution of class assignments using an activation function (e.g., softmax). The activation function may output a vector representing the probability distribution of a list of potential outcomes (i.e., indicating that a defect belongs to a particular class). The defect classification module 520 is configured to assign a class label to a particular defect image based on the output of the activation function. For example, the defect classification module 520 may assign the class label with the highest probability score. In some cases, the defect classification module 520 may set a probability threshold that must be met for a given class assignment to occur. In some cases, the defect classification module 520 may assign a defect class label and include a confidence / probability level.

[0170] The output of the defect classification module 520 includes defect classes linked to defect images 524. If the object detection module 504 assigns a unique defect ID to each defect detected in the inspection image 502 and stores defect data linked to the defect ID, the class assignment may similarly be linked to the defect ID and stored together with the defect ID. The defect classification output may be used by the object detection module 504 to verify the defect class label for the detected object (for example, by comparing the class labels).

[0171] The object detection module 504 and defect classification module 520 in Figure 5 represent a two-stage object detection model of the visual inspection system 300. A representation of the two-stage object detection model according to one embodiment is shown in Figures 6A and 6B.

[0172] Figure 6A shows the comprehensive two-stage object detection model 600.

[0173] Figure 6B shows a two-stage object detection model 650 for camshaft visual inspection implementation.

[0174] For both models 600 and 650, the first stage of the two-stage object detection model includes object detection (defect detection) performed by the object detection module 504.

[0175] The second stage of the two-stage object detection models 600 and 650 includes defect classification, which is performed by the defect classification module 520 using the output 518 of the object detection module 504.

[0176] In other embodiments, the object detection module 504 may implement a one-stage object detection model.

[0177] The one-stage model is represented in Figures 7A and 7B.

[0178] Figure 7A shows a comprehensive one-stage object detection model 700 for visual inspection.

[0179] Figure 7B shows a one-stage object detection model 750 for camshaft visual inspection implementation.

[0180] For models 700 and 750, the one-stage object detection model performs object detection and defect classification using a single model. In such cases, object detection models 700 and 750 are configured to perform the defect classification function of the defect classification module (i.e., the functionality of the defect classification module is implemented as part of the object detection module).

[0181] The two-stage object detection models 600 and 650 can offer various advantages, such as improved defect detection (via the object detection model) and improved classification of detected defects (via the defect classification model). Furthermore, two-stage objects can improve the identification of false positive defects (for example, "candidate defects" detected by the object detection model but not recognized as defects by the classifier).

[0182] In one embodiment, the two-stage object detection models 600, 650 are modified to include a single defect class (NG) instead of multiple defect classes. This can provide a greater dynamic range for the defects identified and classified, and improve poor inter-class model performance.

[0183] In one embodiment, the defect classification model 524 uses ResNet50 with CUDA and TensorRT (a pruning and optimization library for scaling down the model size). This configuration of the defect classification model can significantly increase the classification speed. Furthermore, this configuration of the defect classification model can easily satisfy the real-time detection requirements for two-stage detection and classification models 600 and 650.

[0184] Referring to Figure 8, a comparison of the classification model performance for steel defects according to this embodiment is shown.

[0185] Figure 8 illustrates the first defect classification model 802 and the second defect classification model 850. The second defect classification model 850 is a modified version of the first classification model 802 that shows improved processing speed. Classification models 802 and 850 are configured to analyze input images containing defects, such as defect image 524, and assign defect class labels to the input images.

[0186] The defect classes include entrapment scale 804, patch 806, crazing 808, uneven surface 810, inclusion 812, and scratch 814.

[0187] Examples of input images are shown for the following defect types: entrapment scale defect type 816, patch defect type 818, crazing defect type 820, uneven surface defect type 822, inclusion defect type 824, and scratch defect type 826.

[0188] The second defect classification model 850 has been modified to include ResNet50, CUDA, and TensorRT. TensorRT accelerates the performance of model 850 compared to model 802. TensorRT serializes model 850 to a memory and GPU-efficient engine so that it runs at real-time speeds. TensorRT-based applications (e.g., model 850) can run up to 40 times faster during inference than on CPU-only platforms. TensorRT optimizes neural network models trained in major frameworks, calibrating low accuracy to high accuracy, and can be deployed to hyperscale data centers, embedded, or automotive product platforms. TensorRT is built on CUDA and leverages libraries, development tools, and technologies in CUDA-X to enable inference optimization for deep learning frameworks implemented by the systems described herein. TensorRT provides INT8 and FP16 optimizations for production deployment of deep learning inference applications. Reduced precision reasoning significantly reduces application latency, which can be particularly important in the development of visual inspection applications.

[0189] The first defect classification model 802 showed an average speed of 100ms per "200 x 200" pixel input image.

[0190] The second defect classification model 850 showed an average speed of 3.34 ms per "200 x 200" pixel image.

[0191] As shown, the second defect classification model 850 has significantly improved defect classification speed and performance. The improved defect classification speed and performance reduces inspection time. Reducing inspection time ensures that the visual inspection system does not fall behind the cycle time of the manufacturing line. The cycle time of the inspection system is a major design factor and depends directly on the speed of the AI ​​algorithm and model. In this particular example, the second classification model 850 showed a speedup of 2666% compared to the first classification model 802. Thus, various embodiments of the object detection components (particularly the defect classification components) described herein may be subjected to similar modifications to the classification model architecture to achieve speed improvements and enable real-time defect detection (often required in manufacturing quality inspection). Referring again to Figure 5, the inspection image 502 is also provided for the golden sample module 526.

[0192] Generally, the golden sample module 526 is configured to analyze the inspection image 502 and the golden sample image, detect differences between the images (such differences may be expressed as “objects,” “items,” “candidate regions,” or “(candidate) features,” and may be described by location data indicating the location of the detected differences), and record the differences as an output (e.g., an annotated inspection image). The output of the golden sample module 526 can be used, along with the output 518 of the object detection module 504 (and potentially the output of the defect classification module 520), to detect anomalies and new defects. New defects are considered to be defects that exist in the inspection image 502 but were not recognized by the object detection module 504 (but were identified by the golden sample module 526).

[0193] The Golden Sample Module 526 uses the Golden Sample Image 528. The Golden Sample Image 528 (or "Golden Sample," or "Golden Image") is a reference image intended to represent a "perfect" or "clean" version of the Inspection Image 502. In other words, the Golden Sample Image 528 is considered perfect by the system. However, the Golden Sample 528 only needs to represent a perfect image for the purpose of analysis by the Golden Sample Module 526; it does not necessarily have to be perfect. In other words, the Golden Sample Image 528 is intended to provide a reference representing a perfect or clean image of the target item acquired at the same location as the Inspection Image 502.

[0194] The golden sample module 526 receives the inspection image 502. The inspection image 502 may be received from the camera 304 or the object detection module 504.

[0195] The golden sample module 526 passes the inspection image 502 to the generation model 530. The generation model 530 may be an autoencoder such as a variational autoencoder. The generation model 530 is configured to generate a golden sample image 528 from the inspection image 502. The golden sample image 528 generated using the generation model 530 may be considered a "generated golden sample". In other embodiments, a non-generated golden sample image may be used by the golden sample module 526.

[0196] In embodiments of the autoencoder, the generation model 530 may include an encoder component, a code component, and a decoder component. The encoder component compresses the input (test image 502) to generate a code component. The decoder component reconstructs the input using only the code component. In this case, the term “reconstruct” refers to reconstructing a less noisy representation of the test image 502 than the test image 502 itself. Preferably, the reconstructed representation is noise-free. The reconstructed representation is a golden sample image 528 for a given test image 502 used as input to the generation model 530.

[0197] The generative model 530 may include an encoding method, a decoding method, and a loss function for comparing the output with a target.

[0198] Generally, the generative model 530 receives an inspection image 502 containing a mixture of noisy and non-noisy data. The generative model 530 may be trained using a training process that helps set the model's weights so that it can distinguish between noise and data. Once trained, the generative model 530 will generate and output a non-noisy (or less noisy) image (i.e., a golden sample image 528) even when a noisy inspection image 502 is sent to it. The noise to be removed from the inspection image 502 may be any other deviation from the machined surface that is considered a defect or standard. The noise present in the inspection image 502 and configured for the generative model 530 to remove may have various sources. The noise may be, for example, different types of defects or exceptional objects, droplets (of liquid) or stains from such liquids. Generally, the environment and air in factories and other manufacturing facilities are not clean, and as a result, metal fragments, coolant residue, oil droplets, etc. may be left on the target article 306 (e.g., camshaft) after machining (e.g., CNC machining) and remain on the floor for extended periods. Furthermore, the target article 306 may be washed or covered with protective materials such as rust inhibitors or oils.

[0199] In one embodiment, instead of learning arbitrary functions (for example, as in the case of a standard autoencoder), the generative model 530 learns the parameters of a probability distribution that models the input data. The generative model 530 samples points from the distribution.

[0200] The golden sample module 526 also passes the inspection image 502 to the first pre-trained CNN 532. The first pre-trained CNN 532 generates an inspection image feature map from the inspection image 502.

[0201] The Golden Sample module 526 passes the Golden Sample image 528 to the second pre-trained CNN 534. The second pre-trained CNN 534 generates a Golden Sample feature map from the Golden Sample image 528.

[0202] The golden sample module 526 then uses feature map analysis 536. Feature map analysis 536 includes comparing the inspection image feature map and the golden sample feature map to identify differences in the feature maps. The golden sample module 526 saves the differences as feature map differences.

[0203] The golden sample module 526 may identify feature map differences by characterizing the differences as regions of interest and defining these regions of interest using bounding boxes. Bounding box data (coordinates) for the differences may be linked to the saved and inspected image 502. In one embodiment, the golden sample module 526 generates an annotated output image that includes the inspected image 502 and bounding boxes that identify regions of the inspected image 502 corresponding to the feature map differences identified via the feature map analysis 536.

[0204] The golden sample module 526 passes the feature map difference to the centroid and shape analysis component 538.

[0205] The centroid / shape analysis component 538 analyzes the binary image generated by applying thresholds to the feature map to find the centroid and boundary pixels of those clusters. The golden sample module 526 is configured to generate a result output (golden sample output) containing the pixel coordinates of the centroid for such clusters.

[0206] Referring to Figures 9A and 9B, examples 900 and 950 of representations of a variational autoencoder, which may be implemented by a golden sample module 526 or the like according to one embodiment, are shown.

[0207] Figure 9A shows a first representation example 900 of the variational autoencoder. The autoencoder can be used to generate a reconstructed output image 902 from the original input image 904. The input image 904 may be the test image 502 in Figure 5, and the output image 902 may be the golden sample image 528 in Figure 5.

[0208] The autoencoder includes an encoder 906, a latent representation 908, and a decoder 910.

[0209] As shown, the original input image 904 is provided to the encoder 906. The encoder 806 is configured to generate a latent representation 908 of the input 904. The latent representation 908 is provided to the decoder 910. The decoder 910 is configured to generate an output 902 reconstructed from the latent representation 908.

[0210] Figure 9A shows a second representation example 950 of the variational autoencoder.

[0211] The autoencoder includes an input layer 952, a first hidden layer 954, a code layer 956, a second hidden layer 958, and an output layer 960. The input layer 952 and the first hidden layer 954 form the encoder unit 962 of the autoencoder. The second hidden layer 958 and the output layer 960 form the decoder unit 964 of the autoencoder.

[0212] Similar to the autoencoder in Figure 9A, an input image (e.g., a test image 502) may be provided to the encoder unit 962 via the input layer 952. The encoder unit 962 generates a post-representation of the input image in the code layer 956. The latent representation is provided from the code layer 956 to the decoder unit 964, which generates an output (e.g., a golden sample image 528) reconstructed from the latent representation of the input.

[0213] The variational autoencoder 950 may also be an autoencoder in which training is standardized to avoid overfitting and which ensures that the latent space has good properties that enable the generation process.

[0214] Like standard autoencoders, the variational autoencoder 950 is an architecture that includes an encoder 962 and a decoder 964, which are trained to minimize reconstruction errors between the data being encoded / decoded and the initial data.

[0215] In a variational autoencoder, the input may be encoded as a distribution across a latent space. This introduces some rules of the latent space. The variational autoencoder 950 may be trained as a distribution across the latent space with a first encoding input. Next, points from the latent space are sampled from the distribution. Next, the sampled points are decoded and the reconstruction error is calculated. Next, the reconstruction error is backpropagated through the network. However, a slight modification to the encoding / decoding process is required to introduce some rules of the latent space; that is, instead of the input being encoded as a single point, it is encoded as a distribution across the latent space.

[0216] Referring to Figure 10, a golden sample processing and analysis pipeline 1000 according to one embodiment is shown. The pipeline 1000 may be implemented by the golden sample module 526 shown in Figure 5.

[0217] Pipeline 1000 begins with inspection image 1002 of the target item.

[0218] The inspection image 1002 is passed to the generation model 1004. In pipeline example 1000, the generation model 1004 is a variational autoencoder, such as the variational autoencoder shown in Figures 9A and 9B. The generation model 1004 is configured to generate a golden sample image 1006 from the inspection image 1002 (i.e., the generated golden sample image).

[0219] The generative model 1004 includes a component 1008 which includes at least one hidden layer of the encoder unit of the generative model and a code layer containing encoded data or latent representation.

[0220] Generally, the test image 1002 is passed to the input layer of the generative model 1004, and a reconstructed output containing the golden sample image 1006 is generated at the output layer of the generative model 1004.

[0221] The test image 1002 and the golden sample image 1006 are passed to the golden sample analysis component 1010, respectively. Generally, the golden sample analysis component 1010 is configured to detect and identify the difference between the test image 1002 and the golden sample image 1006.

[0222] The golden sample image 1006 is supplied to the first pre-trained convolutional neural network 1012, and a golden sample feature map is generated.

[0223] The inspection image 1002 is supplied to the second pre-trained convolutional neural network 1014, and an inspection image feature map is generated.

[0224] The golden sample analysis component 1010 compares the golden sample feature map and the inspection image feature map to generate a feature map comparison 1016. The feature map comparison 1016 shows instances of differences between features. Such instances of differences may be represented as an annotated inspection image that includes indicators of the locations where differences were detected. This may include generating bounding boxes that define the locations of the differences.

[0225] Generally, the differences identified between the inspection image 1002 and the golden sample image 1006 can be broadly categorized into two categories: defects and anomalies. Defects may correspond to defects identified by the object detection module 504 (previously learned or known defects) or to objects not identified by the object detection module 504 ("new" defects). Classification of the differences between the inspection image 1002 and the golden sample image 1006 is performed using the output 518 of the object detection module 504 (and, in some cases, the output of the classification module 520). For example, differences identified by the golden sample module 526 that were also detected by the object detection module 504 (i.e., in the object detection output) may be labeled.

[0226] In 1018, centroid and shape analysis is performed on the differences identified in the comparison image. The output of the centroid and shape analysis may be used to generate a golden sample output, which includes an annotated inspection image that can be compared with the object detection output (e.g., an annotated inspection image with detected defects, an annotated inspection image with detected and classified defects).

[0227] The output of the centroid and shape analysis 1018 may be fed into the anomaly detection module 540 along with the defect detection (and, optionally, defect classification) output image to generate an annotated anomaly detection image 1020. To generate the output image 1020, the object detected in the object detection output is compared with the object detected in the golden sample analysis output (i.e., the difference between the golden sample image 1006 and the inspection image 1002) to determine whether the detected object is a known or previously identified defect, a new (unknown) defect, or an anomaly. The output image 1020 is an annotated inspection image in which each difference is identified and labeled by a bounding box. Examples of previously learned defects 1022, newly learned defects 1024, and anomalies 1026 are shown in the output image 1020.

[0228] In other embodiments of pipeline 1000, a generative model component 1008 may be used instead of the golden sample analysis component 1010. In such an embodiment, the golden sample image 1006 and the inspection image 1002 may be supplied to the generative model component 1008, respectively, and their respective latent representations may be generated. The latent or encoded representations may then be compared to determine the difference between the golden sample image 1006 and the inspection image 1002. The difference is recorded by the golden sample module and passed to the anomaly detection module 540.

[0229] Referring to Figure 11, the golden sample module 526 may learn the distribution of anomalies using a semi-supervised or unsupervised learning algorithm.

[0230] For example, in one embodiment, the golden sample module 526 may use distance-based anomaly detection 1102. A plot 1104 is shown containing golden sample image data 1106 and a test image 1108. Distance-based anomaly detection 1102 finds anomaly data based on the distance 1110 between the golden sample image and the new data (i.e., the test image 1108).

[0231] In other embodiments, the golden sample module 526 may use unsupervised anomaly detection 1112. A plot 1114 is shown containing known types of defective data (i.e., normal data) 1116, noise type data (exceptional data) 1118, and novel types of defective data (exceptional data) 1120. Unsupervised anomaly detection 1112 finds anomalous data based on samples observed before the normal and anomalous data. A divider line 1122, which may be generated after training, is shown around the normal data 1116.

[0232] Refer to Figures 12 to 14 illustrating images generated by the visual inspection system of this disclosure according to an embodiment.

[0233] Refer to Figure 12, which shows an example image 1200 illustrating the performance of the generation golden sample processing of the present disclosure compared to golden sample processing using reference images from a golden sample reference bank.

[0234] Figure 12 shows an inspection image 1202 of the camshaft acquired by a visual inspection system such as the visual inspection system 300 (i.e., the one using the camera 304) shown in Figure 3.

[0235] Image 1204 is a golden sample image obtained using a non-generated golden sample approach, in which the golden sample image is identified from a bank of golden sample images using an image processing technique (e.g., registration-based) that utilizes the inspection image. This version of golden sample analysis may be implemented by the visual inspection system described herein, but it has drawbacks compared to the generated golden sample approach. For example, this approach may be sensitive to variations in the machined surface finish of the target article.

[0236] The ungenerated golden sample image 1204 contains two scratches, 1206 and 1208. The presence of these scratches 1206 and 1208 in golden sample image 1204 may introduce an error into the entire golden sample algorithm.

[0237] Image 1210 is a customized (or generative) golden sample image for inspection image 1202 using the generative golden sample approach of the present disclosure. Image 1210 is generated from inspection image 1202 using a generative model (autoencoder), such as generative model 530. The generative model is trained to recognize noise in inspection image data 1202 so that a noisy inspection image is sent to the autoencoder and the autoencoder generates a noise-free image that can function as a golden sample image for the inspection image. In this case, as understood, the generative golden sample processing advantageously generates a golden sample image 1210 that does not contain the two scratches 1206, 1208 present in the non-generative golden sample image 1204. By doing so, both golden sample analysis and the overall anomaly detection process implemented by the visual inspection system are improved.

[0238] Referring to Figure 13, several examples 1300 of golden sample images 1302, 1304, and 1306 generated using the generating golden sample processing of the present disclosure according to one embodiment are shown. The generating golden sample processing is indicated by arrow 1307. The image shows a camshaft being visually inspected using the visual inspection system of the present disclosure.

[0239] Golden sample images 1302, 1304, and 1306 were generated from inspection images 1308, 1310, and 1312, respectively, using the generated golden sample process 1307.

[0240] As illustrated, inspection image 1308 includes two defects (or noise elements) 1314 and 1316. Inspection image 1310 includes two defects 1318 and 1320. Inspection image 1312 includes two defects 1322 and 1324.

[0241] The generated golden sample process 1307 reconstructed golden sample images 1302, 1304, and 1306 while removing defects from inspection images 1308, 1310, and 1312. The golden sample process 1307 removes noise from inspection images 1308, 1310, and 1312 in order to generate golden sample images 1302, 1304, and 1306.

[0242] Golden sample images 1302, 1304, and 1306, along with their respective test images 1308, 1310, and 1312, can be used for further golden sample analysis (their output being used for anomaly detection).

[0243] Referring to Figure 14, several examples 1400 of golden sample images using non-generating and generating golden sample processing according to the embodiment are shown. The images are of a camshaft being visually inspected using the visual inspection system of the present disclosure.

[0244] Images 1402a and 1402b are examples of non-generated golden sample images for use with the golden sample module 526 using the golden sample reference bank approach. The non-generated process may include the application of image processing techniques, such as registering one or more golden sample images for inspection and determining the closest one for comparison.

[0245] Images 1404a and 1404b are examples of generated golden sample images for use by the golden sample module 526, generated using the generated golden sample processing of the present disclosure.

[0246] The ungenerated golden sample image 1402a includes false-negative detection 1406 and false-positive detection 1408.

[0247] In contrast, the generated golden sample image 1404a contains neither false negatives nor false positives (including those present in image 1402a).

[0248] Images 1402a and 1404a provide analyses from two different versions of the golden sample for the same image. Images 1402b and 1404b are the output feature maps (i.e., the ungenerated golden sample feature map and the generated golden sample feature map) of versions 1402a and 1404a for the same input image.

[0249] Image 1402b is generated using non-generating golden sample technology. Image 1404b is generated using generating golden sample technology. In the case of image 1404b, the generating model is used to synthesize the golden sample image. In the case of 1402b, an actual image of the "OK" target item (camshaft) is used.

[0250] To be understood, image 1402a includes false negative 1406 and false positive 1408. The camshaft in image 1402 includes a rubber seal ring in this segment, and its joint (the area in the red box called "false positive") may come into contact with something around the camshaft during installation. The synthetic golden sample 1404a can generate this crack in exactly the same location as in the inspected image. The reason for the false positive 1408 is that the non-generated golden sample technique cannot account for positional variability while performing a pixel-by-pixel comparison of the inspected image and the actual image of the "OK" portion which has a joint that may differ from the inspected image.

[0251] Due to these challenges, the non-generated golden sample technique has low sensitivity to minute anomalies such as clusters of small defects. This also explains why it failed to detect the patch of dents across the camshaft (i.e., false negative 1406).

[0252] Referring again to Figure 5, pipeline 500 further includes an anomaly detection module 540.

[0253] The anomaly detection module 540 receives and processes the output from the object detection module 504 and the golden sample module 526.

[0254] In some cases, the anomaly detection module 540 may receive defect class information determined by the defect classification module 520.

[0255] Thus, in pipeline 500, the object detection module 504, the defect classification module 520, and the golden sample module 526 are each shown to be connected to communicate with the anomaly detection module 540. However, in certain implementations, it is understood that only either the object detection module 504 or the defect classification module 520 may pass information to the anomaly detection module 540.

[0256] In one embodiment, an anomaly detection module 540 is implemented on the same device that implements the object detection module 504.

[0257] The anomaly detection module 540 receives a golden sample output from the golden sample module 526, which includes an annotated inspection image. The annotated inspection image contains detected items, which have differences between the golden sample image 528 and the inspection image 502 identified during the feature map analysis 536. The detected items may be described using bounding boxes that define regions of interest containing the detected differences.

[0258] The anomaly detection module 540 receives object detection output (e.g., object detection output 518) from the object detection module 504, which has one or more bounding boxes, each containing a defect detected by the object detection module 504. The object detection output received by the anomaly detection module 540 may further include defect classification data (i.e., defect class labels) generated by the defect classification module 520.

[0259] The anomaly detection module 540 analyzes the golden sample output and the object detection output to identify differences between the outputs. This analysis may include a comparison of annotated inspection images. For example, each of the golden sample output and the object detection output may include an inspection image 502 annotated with various metadata or annotation data, such as bounding boxes that define regions of interest determined by each module 504, 526. The anomaly detection module 540 may compare the annotation data (e.g., bounding box coordinates) to determine whether there is a difference between the golden sample output and the object detection output.

[0260] Items appearing in both the golden sample and the object detection output may be tagged as confirmation defects by the anomaly detection module 540. Since such confirmation defects are detected by the object detection module 504, they may be considered "known defects" or "previously learned defects" (i.e., recognized or learned by the system). Thus, the anomaly detection module 540 provides a double check against the results of the object detection module 504. In some embodiments, the output (and the items or "objects" or "candidate regions" identified therein) does not need to be the same for the anomaly detection module 540 to tag an item as a confirmation defect. For example, the anomaly detection module 540 may be configured to accept and confirm items having substantially the same location, which may be determined by an acceptable pixel difference. In other embodiments, the items do not need to have the same location for the anomaly detection module 540 to accept or confirm a defect.

[0261] In one embodiment, the anomaly detection module 540 is configured to compare object location and class information provided by the golden sample module 526 and the object detection module 504 in order to determine whether modules 504 and 536 have detected the same object. The anomaly detection module 540 may compare only object location and class information.

[0262] As shown above, the anomaly detection module 540 may be configured to be tolerant of variations. In a particular embodiment, the anomaly detection module 540 determines whether the centroid of a candidate anomaly region (object) from the golden sample output lies inside the object bounding box from the object detection output. If the centroid lies inside the object bounding box, the anomaly detection module 540 identifies the object as defective. If the centroid lies outside the object bounding box, the anomaly detection module 540 may identify and flag the object as an anomaly.

[0263] The difference between the golden sample and the object detection output may be flagged by the anomaly detection module 540 and presented to the user (e.g., an expert) for review. This may include presenting the difference via an annotated inspection image, including the difference, in a user interface displayed on the user device. Items present in the golden sample output but not in the object detection output may be new defect types (i.e., defect types not detected by the current object detector model 508) or anomalies (non-defect variations or artifacts).

[0264] Items detected by the golden sample module 526 but not by the object detection module 504 (i.e., differences between the two outputs) may be presented in the user interface. For example, the user interface may present an annotated version or portion of the inspection image 502 containing details about the difference and request user input to identify the new item (i.e., the difference) as a new defect type or anomaly. The user can view the annotated inspection image in the user interface and provide input data to identify the item as an anomaly or new defect. In the case of a new defect type, the user interface may be configured to present a user interface element for receiving user input data with a defect class label indicating the defect type.

[0265] The anomaly detection module 540 may then label or tag newly detected items as anomalies or new defect types based on input data received from the user, and may store this information in the system. This may include a unique anomaly identifier for anomalies or a new defect identifier for new defects. Annotated inspection images containing new defect types may be identified or tagged as training samples (which may include label information such as new defect classes). Training samples may be used as training data for further training of object detection models 508 and / or defect classification models 522 so that models 508 and 522 can be updated to detect and / or classify new defect types. Such updates to models 508 and 522 may be performed periodically. The anomaly detection module 540 is configured to send inspection images identified as containing new defect types to a training database for use in subsequent training.

[0266] The anomaly detection module 540 and the entire anomaly detection pipeline 500 can offer various advantages. The anomaly detection module 540 enables the detection of anomalies. The anomaly detection module 540 enables the identification of new defects and, as part of an ongoing process, enables the subsequent training of a model to recognize new defects. The anomaly detection module 540 can also provide a double-check mechanism for the system's object detection 504 and defect classification 520 components that can reduce errors such as false positives.

[0267] In one embodiment of pipeline 500, the object detection module 504 generates an output (e.g., a list of detected objects) that includes the object location (e.g., bounding box coordinates, object location data) and object class for each object detected by the object detection model 505. The classification module 520 generates an output that includes the object (defect) class using the classification model 522. The classification module 520 may then supply object class information to the object detection module 504. The object detection module 504 compares the object class from the object detection model 505 with the object class from the classification model 522 to determine whether the class assignments match. If the class assignments of modules 504 and 520 match, the object detection module 504 passes the object class information and object location information to the anomaly detection module 540 for comparison with the golden sample output.

[0268] Refer to Figures 15 and 16, which illustrate anomaly detection pipelines 1500 and 1600 for camshaft visual inspection according to the embodiment.

[0269] Anomaly detection pipelines 1500 and 1600 are similar to anomaly detection pipeline 500, described with reference to Figure 5, with the exception of differences discussed below, including their application to camshaft visual inspection. To avoid repetition, similar content will not be discussed. Therefore, all content described for pipeline 500 may also apply to pipelines 1500 and 1600, except for the differences discussed below.

[0270] In Figures 15 and 16, elements and components having similar functions to those of pipeline 500 in Figure 5 are represented by similar reference numbers with 1000 and 1100 added, respectively (i.e., reference numbers from the 1500 and 1600 series are used). In other words, if an element or component of pipeline 500 is represented by reference number 5xx, and that element or component has a counterpart in pipeline 1500 or pipeline 1600, then the elements or components of the counterpart in pipeline 1500 and 1600 are represented by reference numbers 15xx and 16xx, respectively.

[0271] Referring to Figure 15, the anomaly detection pipeline 1500 begins with inspection image 1502. Inspection image 1502 is the camshaft at a specific position on the camshaft.

[0272] The inspection image 1502 is supplied to the object detection module 1504. The object detection module 1504 is implemented on the first node computing device (Jetson 1). The object detection module 1504 is configured to process the inspection image 1502, which has a pixel resolution of "800 x 600". The object detection module 1504 may process the inspection image 1502 in 40 ms.

[0273] The object detection module 1504 includes a CNN comprising an object proposal component 1506, a feature map component 1508, an ROI pooling component 1510, a fully connected layer component 1512, a softmax component 1514, and a regressor component 1516.

[0274] The object detection module 1504 uses a CNN to generate an object detection output image 1518. In this case, the object detection output image 1518 includes the detected defect 1519. The detected defect 1519 is identified by its bounding box in the output image 1518. The output image 1518 is annotated with metadata (e.g., bounding box information) describing the detected defect 1519.

[0275] The output image 1518 is passed from the object detection module 1504 to the anomaly detection module 1540.

[0276] The output image 1518 may also be supplied to a defect classification module (not shown), such as a defect classification module 520 or 1620. Supplying the output image 1518 to a defect classification module may include supplying the entire output image 1518 or a portion specifically corresponding to the detected defect 1519 (i.e., the region of interest containing the defect 1519). If only the portion containing the detected defect 1519 is passed to the defect classification module 1520, the region of interest may be effectively cropped from the output image 1518 using bounding box information or the like.

[0277] The inspection image 1502 is also passed to the golden sample module 1526. The golden sample module 1526 is implemented on a second-node computing device (Jetson 2). The golden sample module 1526 is configured to process the inspection image 1502 and the golden sample image 1528, which have a specific pixel resolution (e.g., "100 x 300pi"). The golden sample module 1526 may process the inspection image 1502 and the golden sample image 1528 in 36ms. The CNN architecture for the object detection module 1504 may differ from that of the golden sample module 1526. Since the input size requirements for the two networks are determined by the network design, in some cases the input image may be resized by one or both modules 1504 and 1526 to match the input requirements of each network.

[0278] The golden sample module 1526 passes inspection image 1502 and golden sample image 1528 to the first and second pre-trained CNNs 1532 and 1534, respectively, to generate feature maps for inspection image 1502 and golden sample image 1528. The feature maps are compared using feature map analysis 1536. Feature map analysis 1536 shows an example of an output image that may be generated by feature map analysis 1536. As can be seen, certain differences (detected items) are detected. The output of feature map analysis 1536 is provided to the centroid and shape analysis component 1538.

[0279] The golden sample image 1528 may be a generated golden sample image generated by a generation model such as the generation model 530 in Figure 5. In other embodiments, the golden sample image 1528 may be a non-generated golden sample image. For example, the golden sample image 1528 may be retrieved from a reference bank of golden sample images based on an image registration process. This may include identifying the location where the inspection image 1502 was acquired (e.g., the angle or position of the camshaft rotation) and performing image registration of one or more candidate golden sample images from the reference bank to identify the golden sample image 1526 for use by the golden sample module 1526.

[0280] The output of the centroid and shape analysis component 1538 is passed from the golden sample module 1526 to the anomaly detection module 1540.

[0281] In some cases, only a subset of the data may be passed from modules 1504 and 1526, respectively, to the anomaly detection module 1540. For example, object detection module 1504 and golden sample module 1526 may send only the information necessary to perform a comparison between defects detected by object detection module 1504 and detected items identified by golden sample module 1526. This information may include, for example, bounding boxes or other image metadata annotating images (e.g., size, defect class).

[0282] Upon receiving the object detection image 1518 and the output of the golden sample module 1526, the anomaly detection module 1540 performs a comparison on the output. This involves generating an annotated inspection image 1542 that includes metadata describing the objects detected by the object detection module 1504 or the golden sample module 1526. Objects may be tagged with identifiers depending on whether the object was determined to be a known defect, and whether objects other than known defects are new defects or anomalies.

[0283] Annotated image 1542 includes objects 1544, 1546, and 1548.

[0284] Object 1544 corresponds to the detected defect 1519 identified by the object detection module 1504 and is confirmed via the golden sample output.

[0285] Objects 1546 and 1548 were not detected by the object detection module 1504 (as indicated by the absence of bounding boxes at their locations in image 1518), but were identified by the golden sample module 1526. Therefore, the anomaly detection module 1540 flags objects 1546 and 1548 for review by a human expert.

[0286] As previously mentioned, objects 1546 and 1548 may be new defect types or anomalies. The anomaly detection module 1540 triggers the presentation of the annotated image 1542 (or the portion containing objects 1546 and 1548 or its metadata) via the user interface. The user interface is configured to receive input data from a user who has labeled object 1546 as a new defect or anomaly. In this case, object 1546 is labeled as a new defect type, and object 1548 is labeled as an anomaly.

[0287] For training or retraining the object detection module 1504 to detect new defect types, when object 1546 is labeled as a new defect type (for example, via user input), the anomaly detection module 1540 may be configured to send the portion containing image 1542 or object 1546 to the object detection training database.

[0288] Referring to Figure 16, the anomaly detection pipeline 1600 operates in substantially the same manner as pipeline 1000. Pipeline 1600 further includes a defect classification module 1620. The defect classification module 1620 is implemented on the same node device (Jetson 1) as the object detection module 1620. The defect classification module 1620 processes images at a pixel resolution of "200 x 200". The defect classification module 1620 can process images in 4 ms.

[0289] The defect classification module 1620 receives a defect image 1624 from the object detection module 1604. The defect image 1624 contains the detected defect 1619 identified by the object detection module 1604.

[0290] The defective image 1624 is passed to the classification model 1622, which has a CNN.

[0291] The classification model 1622 analyzes the defect image 1624 and generates a class label for the defect (defect class).

[0292] The system may be configured to perform a certainty check using an object detection network and a classification network. In one embodiment, the object detector passes "object position + object class" to the classification network to ensure that the classification network is viewing the same object class. The certainty check mechanism implemented by the system provides certainty that two different networks (object detection and classification) determine the same defect class at the same relative position. If there is a discrepancy between the classes generated by the two networks, the determination should not remain uncertain. Uncertain detections may be flagged by the system (e.g., via the object detection module 1604) for future investigation, and the system may proceed with the process.

[0293] The defect class information 1625 and the object detection output 1618 are passed to the anomaly detection module 1640 for analysis against the output of the golden sample module 1626. The information sent by the object detection module 1604 and / or the classification module 1620 may include defect class and certainty check information.

[0294] Referring to FIG. 17, a graphical representation 1700 of various embodiments of an anomaly detection algorithm provided by the present disclosure is shown. The discussion below proceeds with reference to specific performance characteristics.

[0295] Four embodiments (Embodiment 1, Embodiment 2, Embodiment 3, and Embodiment 4) are shown. In each of the embodiments, the anomaly detection algorithm includes an image acquisition stage, an object detection stage, and a golden sample stage.

[0296] Graphical representation 1700 includes a time axis 1702 whose start time is equal to "0", and on which the speed performance of the anomaly detection algorithms of the first to fourth embodiments is illustrated.

[0297] Embodiment 1 includes an image acquisition stage 1704, an object detection stage 1706, and a golden sample stage 1708. Embodiment 1 further includes a main processing thread 1710 executed on a node computing device (Jetson 1). The image acquisition 1706, object detection 1708, and golden sample 1710 stages are executed sequentially in the main thread 1710 as part of a serialized process. The image acquisition stage 1704 takes 25 ms, the object detection stage 1706 takes 40 ms, and the golden sample stage 1708 takes 62 ms. The total time of the anomaly detection algorithm is 127 ms per image.

[0298] Embodiment 2 includes an image acquisition stage 1712, an object detection stage 1714, and a golden sample stage 1716. Embodiment 2 includes a modification to the golden sample stage 1708 of Embodiment 1. Embodiment 2 also includes a main processing thread 1718 executed on a node device (Jetson 1). The optimization of the golden sample reduced the time of the golden sample stage 1716 from 62 ms to 36 ms, and reduced the total processing time of the anomaly detection algorithm from 127 ms to 101 ms per image. This provided a +25% improvement in execution speed.

[0299] Embodiment 3 includes an image acquisition stage 1720, an object detection stage 1722, and a golden sample stage 1724. Embodiment 3 of the anomaly detection algorithm further includes parallelization of the anomaly detection algorithm of Embodiment 2. Embodiment 3 includes a first main processing thread 1726 running on a first node device (Jetson 1), a child processing thread 1728 running on the first node device, and a second main processing thread 1730 running on a second node device (Jetson 2). The first main processing thread 1726 includes the image acquisition stage 1720 and the object detection stage 1722. The second main thread 1730 includes the golden sample stage 1724. Optimizations that result in time reduction include implementation of the code in other programming languages. The parallelization of the anomaly detection algorithm in Embodiment 3 reduced the total processing time per image from 101ms to 65ms (providing a +55% improvement in execution time speed).

[0300] In one embodiment, a first main thread running on Jetson 1 (first node device) executes an image acquisition and object detection network. The first main thread creates a child thread to send a test image to Jetson 2 (second node device with a golden sample module running the second main thread) and waits for a response. A "wait" occurs in the child thread, allowing the first main thread to continue costly calculations until it is ready for a response from the child thread (i.e., what the reply from Jetson 1 was). In this case, "wait" is the child thread waiting for a reply from Jetson 2. "Send" is the child thread sending the image to Jetson 2. "Receive" is the child thread receiving the response from Jetson 2. "Thread merge" is the first main thread requesting the child thread to provide the response data before closing the thread to free its memory. "Send" and "Receive" are the encoding and decoding operations of the transferred image using the H264 compression codec.

[0301] Embodiment 4 includes an image acquisition stage 1732, an object detection stage 1734, a defect classification stage 1736, and a golden sample stage 1738. Embodiment 4 is a modification of Embodiment 3 that uses a 120 FPS camera and a two-stage object detection architecture including an object detection stage 1734 and a defect classification stage 1736. Embodiment 4 includes a first main processing thread 1740 on a first node device, a child processing thread 1742 on a first node device, and a second main processing thread 1744 on a second node device. The first main processing thread 1740 includes the image acquisition stage 1732, the object detection stage 1734, and the defect classification stage 1736. The second main processing thread 1744 includes the golden sample stage 1738.

[0302] The camera improvements in Embodiment 4 resulted in improved acquisition time. The previous embodiment used a 41 FPS camera. With a 41 FPS camera, it takes 24.3 milliseconds to acquire image information from the sensor to the software. A 120 fps camera can perform the same process in just 8.33 milliseconds. Over thousands of images, this extra time begins to accumulate. Thus, as much time as possible in the image acquisition stage of the entire pipeline can provide more time for pre-processing / inference / post-processing of the acquired data.

[0303] As illustrated, Embodiment 4 includes an additional classification stage 1736 in the first main processing thread 1740 as part of a two-stage object detection architecture. The classification stage 1736 showed a processing time of 4 ms per image. The image acquisition stage 1732 of the first main processing thread 1740 showed a processing time of 10 ms per image, a reduction from 25 ms per image in Embodiment 3. Embodiment 4 showed a total processing time of 54 ms per image, a reduction from 65 ms per image in Embodiment 3. As a result of the modifications, Embodiment 4 showed a +20% improvement in execution time speed compared to Embodiment 3.

[0304] Embodiments of the visual inspection systems and methods described herein that utilize object tracking are described below.

[0305] Referring to Figure 18, a computer system 1800 for object tracking during visual inspection of a target article, according to one embodiment, is shown.

[0306] The computer system 1800 may be implemented in any one or more computing devices described herein. For example, in certain embodiments, the computer system 1800 may be implemented in the worker node device 310 in Figure 3 or the AI ​​visual inspection device 12 in Figure 1. The computer system 1800 communicates with an object detection module, such as the object detection component 314 in Figure 3. The computer system 1800 may also communicate with the object detection module 504 and the defect classification module 520 in Figure 5.

[0307] The object tracking process implemented by computer system 1800 may offer the advantage of making object detection more robust against false positives.

[0308] The computer system 1800 includes a processor 1802 that communicates with memory 1804.

[0309] The processor 1802 includes an object tracker module 1806. The object tracker module 1806 provides computer-executable instructions that, when executed by the processor 1802, cause the computer system 1800 to perform object tracking processing. Object tracking processing includes tracking an object detected across multiple images or frames of a target article using a computer-implemented object detection process. For example, the target article may be imaged at multiple times while being driven relative to a camera (e.g., a video with multiple frames). In one embodiment, the object being tracked is a defect in the target article detected by an object detection component of a visual inspection system (e.g., object detection component 314 in Figure 3).

[0310] The object tracker 1806 may provide a human operator with confidence that a detected object is not false positive by tracking the detected object over a threshold number of frames. In this way, the object tracker 1806 can advantageously reduce instances of false positive object detection in the visual inspection system.

[0311] Memory 1804 stores the tracked object data structure 1808. The tracked object data structure 1808 contains various data about the tracked object. The data stored in the tracked object data structure 1808 is generated and updated by the object tracker module 1806.

[0312] Data structure 1808 contains a unique object ID 1810, which is assigned by object tracker 1806.

[0313] The data structure 1808 also includes various other data linked to a unique object ID 1810 about the tracked object, including the object start position 1812, current centroid 1814, class 1816, disappearance frame 1818, appearance frame 1820, true detection status 1822, average size 1824, centroid array 1826, motion vector 1828, count value 1830, and approved detection status 1832.

[0314] The count value 1830 represents the count of each specific ID that the object tracker counts each time it passes through a "line" or specific segment in the image. If the object tracker determines and confirms a true detection, it maintains a record of the number of defects observed by the system over the past 24 hours (or any other period). Thus, it is advantageous for the object tracker to count each defect and for the count field (count value 1830) to inform the object tracker that a particular defect has been counted by an external software application.

[0315] In various embodiments, the tracked data structure 1808 may include at least one of the various data types 1812, 1814, 1816, 1818, 1820, 1822, 1824, 1826, 1828, 1830, and 1832. Additional data types may be included and linked to object IDs.

[0316] The object tracker module 1806 may include a software library. The software library may be input into the code base of one or more software components or into a module described in this disclosure. For example, the software library may be input into the code base of an object detection component, such as the object detection component 314 in Figure 3. The software library may also be input into the code base of a golden sample component, such as the golden sample component 316 in Figure 3.

[0317] FIG. 19 illustrates an example code representation 1900 of an object tracker according to one embodiment.

[0318] The object tracker module 1806 takes an initial set of object detections and generates a unique ID for each of the initial detections. The initial set of object detections may be an input set of bounding box coordinates. The initial set of object detections may include a class label for each detected object. The initial set of object detections is provided by an object detection network, such as the object detection component 316 in FIG. 3.

[0319] The object tracker module 1806 tracks each of the objects as they move across subsequent images (frames in a video). The object tracker 1806 maintains the assignment of unique IDs during tracking. In this process, the object tracker module 1806 may use a simple Euclidean distance metric. By using this metric, the object tracker module 1806 can track each object and "maintain" all organized structures that the object tracker module 1806 is observing.

[0320] The object tracker module 1806 generates a unique ID and assigns it to each tracked object (e.g., object ID: 1). The assignment of unique IDs by the object tracker 1806 enables the counting of unique objects in a video. The counting function of the object tracker module 1806 allows for trained, intelligent decisions about each object. Each unique object only needs to be acknowledged once. Unique objects can be counted over time. When an object rotates, its size can be appropriately adjusted across multiple frames. The object tracker module 1806 monitors the motion vectors of objects. Tracking unique objects is a key component of the object tracker 1806.

[0321] The object tracker module 1806 tracks each unique object simultaneously. In doing so, the object tracker module 1806 may use location information (e.g., position and coordinate information in the image frame) and class information (e.g., a defect class identifier such as "scratch") about the object.

[0322] The object tracker module 1806 can track objects that randomly disappear from frames. The algorithm implemented by the object tracker module 1806 expects an object to move the maximum number of pixels in a particular direction each frame. Based on this information, if an object is not observed in the next frame, the object tracker module 1806 marks the object as having disappeared in its dictionary and continues processing. If the object reappears within two frames, the object tracker module 1806 verifies that the Euclidean distance remains within the boundaries of the object's expected position, removes the disappearance flag, and continues normal processing.

[0323] The object tracker module 1806 may use dynamic averaging sizing for each object across multiple frames. Because the intersection over union (IOU) of the object detector when approximating the bounding box around an object is not 100% accurate, the object tracker module 1806 may record the size of the defect across multiple frames to achieve better measurement accuracy. The use of dynamic averaging sizing can provide better accuracy. For example, the bounding box output by the software components of this disclosure configured to detect objects (e.g., the object detector module, the golden sample module) may contain noise and may not accurately reproduce the contour of the detected object. By determining and averaging the object size across multiple frames, the object tracker 1806 may determine the average object size, which is a more accurate or refined measurement of the object size. In embodiments where the defect size is used to determine whether or not it exceeds the acceptable defect range, accurate sizing of the defect may be particularly important. This is because whether or not the defect is within an acceptable defect range ultimately determines whether or not the target article being visually inspected is classified as defective. Thus, inaccurate sizing can lead to some defective target items being accepted, while some non-defective items being mistakenly identified as defective and discarded.

[0324] The object tracker module 1806 may track motion vectors for each object. These motion vectors are used by the object tracker module 1806 to predict the future position where the object is expected to appear in the next frame.

[0325] The object tracker module 1806 may update the tracked object data structure 1804 with the new object at a speed of approximately 0.113 ms. The object tracker module 1806 may be configured so that all existing bounding boxes are correlated with objects already observed and tracked by the object detector prior to the new object. Thus, the remaining bounding boxes for the new object are added as new entities in the tracker. The object tracker module 1806 may index and explore the tracker for a particular object at a speed of approximately 0.395 s. Indexing involves exploring and scanning the object tracker for all stored information about the current object. The object tracker module 1806 may be configured to keep the indexing and exploration speed as fast as possible to minimize latency and overhead in the system.

[0326] In one embodiment, the object tracker module 1806 implements a centroid tracking algorithm. The centroid tracking algorithm relies on the Euclidean distance between the centroid of an existing object (i.e., an object already observed by the object tracker 1806) and the centroid of a new object between subsequent images or frames in the video.

[0327] The centroid tracking algorithm may operate on the main assumption encoded in object tracker 1806 that a given object will potentially move between subsequent frames, but the distance between the centroids of that object for frame "n" and frame "n+1" will be smaller than all other distances between other objects.

[0328] Object tracking module 1806 may have optimized code including loops and functions. The code may be "Cynthonized" to provide performance improvements.

[0329] The object tracking module 1806 is configured to assign and maintain a tracking status for each tracked object. The tracking status is stored in the tracked object data structure 1804. The tracking status may be assigned from several possible tracking statuses.

[0330] The tracking status may include a first tracking status corresponding to an unconfirmed detection (or "not true detection").

[0331] The first tracking status is assigned to tracked objects that have been "observed" or detected in fewer than "N" frames, where "N" is an integer greater than "0". "N" represents a threshold for the number of frames in which the tracked object must be observed for it to be considered confirmed or a true detection by the object tracker module 1806. In one embodiment, "N" is "5". In other words, the first tracking status is assigned to objects that have been detected but have not been tracked over the required number of frames. The number "N" may be determined based on the length of time the object is expected to be observed, from the time the object enters a frame until the object exits the other side of the frame. "N" may be between "3" and "5".

[0332] The tracking status may include a second tracking status corresponding to confirmed or true detection.

[0333] The second tracking status is assigned to tracked objects observed or detected in N or more frames. In other words, the second tracking status is assigned to objects that have been detected and tracked over the required number of frames (e.g., N frames).

[0334] The tracking status may include a third tracking status corresponding to an approved confirmed detection (or approved true detection).

[0335] Third tracking is assigned to tracked objects that have achieved second tracking status (i.e., tracked over "N" frames) and have exceeded the defect tolerance stored in memory 1804 or other data storage accessible to the computer system 1800. In other words, third tracking status may be assigned to objects that have been detected, tracked over the required number of frames, and meet a predetermined defect tolerance.

[0336] In one embodiment, the object tracker 1806 sends tracked object data, including at least the object size (e.g., average size 1824), to the PLC device (e.g., PLC320 in Figure 3) for each tracked object to which a second tracking status has been assigned.

[0337] In some cases, such as when a visual inspection system is configured to detect and classify multiple types or classes of defects, the tracked object data sent to the PLC may further include class labels for the tracked objects.

[0338] The PLC stores defect tolerance information for one or more defect classes. The defect tolerance information may include defect class identifiers linked to size tolerances. The PLC may receive tracked object data to determine whether the tracked object exceeds the defect tolerance, and may refer to the stored defect tolerance information using the defect class and defect size information. The result of the tolerance determination may be provided to the object tracker 1806.

[0339] In some cases, if the defect exceeds the acceptable limit, the PLC may be configured to stop the visual inspection of the target item and treat the target item as defective (e.g., by sending a control signal to remove the target item, or by transporting the defective item to a designated area). If the defect does not exceed the acceptable limit, the PLC may be configured to ignore the defect and continue the visual inspection of the target item.

[0340] In other embodiments, some of the defect tolerance storage and processing functions, such as those performed by the PLC described above, may be performed by the computer system 1800 or other computing devices, such as other computing components that are communicably connected to the computer system 1800. For example, the computer system 1800 may store defect tolerance information in memory 1804 to determine whether the defect tolerance has been exceeded, and a software component executed by the processor 1802 may be configured to refer to the stored defect tolerance information using the tracked object data.

[0341] Object tracker 1806 is configured to assign a third tracking status to an object if it is determined that the tracked object has exceeded the object tolerance limit.

[0342] The object tracker 1806 may be configured to stop tracking an object that has been assigned a third tracking status (approved confirmation detection). In this way, the object tracker 1806 does not consider approved confirmation detections (e.g., those approved by the PLC) in future frames.

[0343] As a variation, the object tracker 1806 may remove overhead significantly faster than actual object detectors, may handle cases where the tracked object disappears or moves outside the boundaries of an image or video frame, may be robust to occlusion, and may be able to pick up objects that the object tracker has lost sight of between frames. Optimization of the object tracker is critical in some embodiments of the visual inspection system, as excessive time or memory usage by the object tracker to perform calculations for tracking would diverte time that could be used for other AI algorithms (e.g., object detection, classification, golden sample analysis). Robustness to occlusion may be provided by the object tracker's ability to predict object motion using past motion vectors.

[0344] Object Tracker 1806 may offer various capabilities and advantages. Object Tracker 1806 may track unique objects simultaneously (via class and position). Object Tracker 1806 may be more robust in tracking objects that randomly disappear from frames. Object Tracker 1806 may offer better accuracy in defect sizing by using dynamic average sizing across multiple frames. Object Tracker 1806 may track the motion vector of each tracked object. Object Tracker 1806 may enable the counting of unique objects. Object Tracker 1806 may ignore an object in future frames if it has been marked by Object Tracker 1806 as an approved true detection.

[0345] Referring to Figure 20, a method 2000 is shown according to one embodiment, which uses the object tracker 1806 of Figure 18 during a visual inspection. Method 2000 may be performed for each frame observed or acquired by a camera imaging the target article.

[0346] In 2002, an inspection image of the target item (e.g., a video frame of the target item) is provided as input to the object detection network. The object detection network includes an object detection model for detecting defects (objects) and a classification model for classifying the detected defects.

[0347] In 2004, the object detection network detects and classifies defects in the inspection image.

[0348] In 2006, the object detection network outputs bounding box coordinates and a class label for each defect detected and classified. The bounding box coordinates define the region of interest in the inspection image containing the defect. The class label indicates the type or class of the defect.

[0349] In version 2008, the bounding box coordinates and class label for each defect detected and classified by the object detection network are provided as input to the object tracker 1806.

[0350] In 2010, the object tracker 1806 updates the tracked object data structure 1808 using bounding box and class information generated by the object detection network. This may include the application of a centroid tracking algorithm by the object tracker 1806 to determine whether a given defect is a previously observed (tracked) defect or a newly observed (not yet tracked) defect. This determination may include the use of bounding box coordinates and class label information.

[0351] If the defect is a newly observed defect, the object tracker 1806 assigns a unique ID 1810 to the defect and tracks the defect over subsequent frames. The object tracker generates a tracked object data structure 1808 for the tracked object linked to object ID 1810.

[0352] If the defect is a previously observed (i.e., tracked) defect, the object tracker 1806 updates the tracked object data structure 1808 corresponding to the tracked defect's unique object ID 1810. This may include updating values ​​for at least one of the data types 1814, 1818, 1820, 1822, 1824, 1826, 1828, 1830, and 1832 in the tracked object data structure 1808. For example, this may include calculating a new average size 1824 (dynamic average sizing) of the defect using size information from the current frame. As another example, this may include updating the tracking status of the tracked defect (e.g., unconfirmed, confirmed, confirmed and approved).

[0353] In some cases, method 2000 may be performed for golden sample analysis. In such cases, the inspection image is provided for the golden sample component (e.g., golden sample component 316). The golden sample output may include bounding boxes for each detected object (i.e., the difference between the feature maps of the inspection image and the golden sample image), but may not include class label information. The object tracker 1806 may then update the tracked object data structure 1804 using the bounding box coordinates.

[0354] Refer to Figures 21A to 21D illustrating inspection images processed using object tracking processing according to one embodiment. For example, the image may be processed using the object tracker 1806 and computer system 1800 shown in Figure 18. The inspection image is a camshaft captured across four frames.

[0355] The camshaft is visually inspected by the system and method described herein. The visual inspection includes seal ring assembly detection. Seal ring assembly detection includes detecting and confirming the presence and orientation of the seal rings.

[0356] Seal ring assembly detection is performed on the seal ring. The seal ring is a plastic component mounted on the camshaft. The equipment performing seal ring installation may not install the seal ring on the camshaft, or may install it incorrectly. A visual inspection system and object tracker (e.g., system 1800) may be configured to verify that the seal ring is present and correctly inserted on the camshaft.

[0357] Referring to Figure 21A, a user interface 2100a is shown that displays a first annotated inspection frame processed using object tracking processing using a computer system 1800, etc., according to one embodiment.

[0358] The visual inspection system inspects three regions of interest ("ROIs") ROI12104, ROI22106, and ROI3 (not shown) to detect and confirm the presence and orientation of the seal rings for the first seal ring 2110 and the second seal ring 2112. ROI3 is not present in this particular part of the camshaft. The camshaft may have up to three specific ROIs per section.

[0359] Object tracker 1806 maintains tracked object counts 2114 for three ROIs: 2104, 2106, and ROI3. As can be understood, tracked object count 2114 includes one tracked object in ROI 2104 (corresponding to the first object 2122 described below).

[0360] Furthermore, object tracker 1806 maintains the seal ring tracking status (true / false) for the second and first seal rings 2116 and 2118. The status of the second seal ring 2116 is false. The status of the first seal ring 2118 is also false.

[0361] The object tracker 1806 is configured to assign an unconfirmed detection tracking status (i.e., a false detection) to tracked objects observed in fewer than five frames. Unconfirmed detections may be labeled in the user interface 2100a using a unique color indicator (e.g., a blue indicator). For example, certain information (e.g., defect ID, centroid location) may be displayed as text near the defect or bounding box for the defect, and such text may be colored or highlighted in color.

[0362] The object tracker 1806 is configured to assign a true detection tracking status to tracked objects observed over five frames. True detection may be labeled in the user interface 2100a using a unique color indicator (e.g., a yellow indicator). For example, certain information (e.g., defect ID, centroid location) may be displayed as text near the defect or the bounding box for the defect, and such text may be colored or highlighted in color.

[0363] The object tracker 1806 is configured to assign an approved true detection status to tracked objects that have been observed over five frames and confirmed to be within acceptable limits. Approved true detections may be labeled in the user interface 2100a using a unique color indicator (e.g., a red indicator). For example, certain information (e.g., defect ID, centroid location) may be displayed in text near the defect or bounding box for the defect, and such text may be colored or highlighted in color. As can be understood, the unique color indicator may make unconfirmed detections, true detections, and approved true detections visually distinguishable from each other so that a user viewing the user interface can understand the detection status. Other forms may be used to uniquely identify and visually distinguish unconfirmed detections, true detections, and approved true detections, and text with a unique color indicator may be used as just one example. For example, in other cases, the detection status may be provided in text near the detected defect.

[0364] Oil and dirt element 2120 are ignored.

[0365] Object 122 has been detected in ROI2104. Since Object 122 has been observed for at least "5" frames and is confirmed to be within acceptable limits, Object Tracker 1806 assigns an approved true detection status to Object 122. Object 122 is a tracked defect.

[0366] The second object 2124 (seal ring tracking) is detected in the first seal ring 2110. Since the second object 2124 has been observed in fewer than 5 frames, the object tracker 1806 assigns the status of unconfirmed detected tracking to the second object 2124.

[0367] A third object 2126 may also be observed. The third object 2126 is the second seal ring. Each camshaft has two seal rings. Therefore, if both seal rings are found, two rectangles may be highlighted (for example, using a unique color indicator (e.g., green)) to indicate the confirmed presence of both seal rings (as shown in Figure 21D).

[0368] Referring to Figure 21B, a user interface 2100b is shown that displays a second annotated inspection frame processed using object tracking processing using a computer system 1800, etc., according to one embodiment. The second inspection frame was acquired after the first inspection frame in Figure 21A.

[0369] At this time, the second object 2124 was observed in five frames. The object tracker 1806 assigns a true detection tracking status to the second object 2124. The true detection status for the second object 2124 indicates confirmation of the presence and orientation of the first seal ring (unauthorized true detection). The object tracker 1806 updates the first seal ring status 2118 to "true".

[0370] The presence and orientation of the first seal ring have not yet been confirmed.

[0371] Referring to Figure 21C, a user interface 2100c is shown that displays a third annotated inspection frame processed using object tracking processing using a computer system 1800, etc., according to one embodiment. The third inspection frame was acquired after the second inspection frame in Figure 21B.

[0372] At this point, the second object 2124 was approved. Object tracker 1806 assigns the approved true discovery status to the second object 2124.

[0373] The presence and orientation of the second seal ring 2112 have not yet been confirmed.

[0374] Referring to Figure 21D, a user interface 2100d is shown, which displays a fourth annotated inspection frame processed using object tracking processing with a computer system 1800, etc., according to one embodiment. The fourth inspection frame was acquired after the third inspection frame in Figure 21C. User interface 2100d shows the state after seal ring assembly detection is complete.

[0375] At this time, the third object 2126 was observed in at least five frames. The object tracker 1806 assigns an approved true detection status to the third object 2126. A true detection status for the third object 2126 indicates confirmation of the presence and orientation of the second seal ring 2112. The object tracker 1806 updates the second seal ring status 2116 to "true". This may include highlighting the bounding box for the second seal ring 2112 using a unique color indicator (e.g., green highlight).

[0376] The systems and methods for visual inspection described herein can reduce false-positive defect detection using efficient model training and object tracking. Efficient model training may include at least one of the following: appropriate model annotation and in-training augmentation, early stopping to avoid overfitting, hyperparameter tuning (LR, optimizer, bounding box proposal), appropriate image preprocessing, and data pruning. Object tracking requires that a unique defect be observed across multiple frames before the object tracker alerts the visual inspection system.

[0377] Referring to Figure 22, a visual inspection pipeline 2200 according to one embodiment is shown. The pipeline 2200 is implemented by one or more computing devices, such as the worker node device 310 in Figure 3. The pipeline 2200 performs defect detection and classification and anomaly detection.

[0378] Pipeline 2200 functions similarly to pipeline 500 in Figure 5. Components in pipeline 2200 that perform the same or similar functions as pipeline 500 are indicated using reference numbers with the same last two suffixes (e.g., Inspection Image 2202, Inspection Image 502). Specific features and functions of such components, as previously described with reference to Figure 5, are not repeated here but should be understood to be present.

[0379] Pipeline 2200 begins with input image 2202. The input image is passed to object detection module 2204 and golden sample analysis module 2226.

[0380] The object detection module 2204 includes an object detection model 2205. The object detection model 2205 takes an input image 2202 as input and generates object detection ("OD") output data 2218. The OD output data 2218 includes object data describing each object detected by the object detection model 2205 in the input image 2202. The object data for a detected object may include, for example, object identifier data that uniquely identifies the object, object class data (e.g., class label), object location data (e.g., bounding box coordinates), object confidence level data, or object size data. The OD output data 2218 may also be in the form of an array, a list, or other data structure containing object data for each detected object. The OD output data 2218 may also be in a format directly comparable to the output data (described later) generated by the golden sample module 2226. The OD output data 2218 in a directly comparable format may be generated directly by the object detection model 2205, generated by the object detection module 2204 by converting the output of the object detection model 2205, or generated by the anomaly detection module 2240 using the OD output data 2218 (i.e., by converting the OD output data to an appropriate format).

[0381] In Figure 22, the object detection model 2205 includes an object suggestion component 2206, a feature map component 2208, an ROI pooling layer component 2210, a fully connected layer component 2212, a softmax layer component 2214, and a regressor layer component 2216. In other embodiments, the object detection model 2205, which is a machine learning model trained to detect one or more classes of objects in an input image 2202 and generate object data about the detected objects, may be configured in a different manner.

[0382] OD output data 2218 is provided from the object detection module 2204 to the anomaly detection module 2240.

[0383] As previously mentioned, input image 2202 is also provided to the golden sample module 2226. Input image 2202 is also provided to the image comparison module 2350.

[0384] The input image 2202 is also provided to the generative model 2230. The generative model 2230 receives the input image 2202 as input and generates the golden sample image 2228 as output. The golden sample image 2228 is provided to the image comparison module 2350.

[0385] Upon receiving the input image 2202 and the golden sample image 2228, the image comparison module 2250 performs an image comparison on the input image 2202 and the golden sample image 2228 to generate image comparison output data 2252 that identifies the differences. The differences generally correspond to artifacts that are present in the input image 2202 but not in the golden sample image 2228. Detected artifacts may be considered "objects" or "regions of interest." Detected artifacts are defined by artifact data in the image comparison output data. The artifact data may include location data that identifies the location of the artifact in the input image. The artifact data may include an artifact identifier linked to the artifact location data. The location data may include bounding box coordinates that define the bounding box surrounding the artifact.

[0386] Image comparison output data 2252 is provided for the exceptional defect and anomaly classifier module 2254.

[0387] The exceptional defect and anomaly classifier module 2254 may include an artifact cropping module. The artifact cropping module receives image comparison output data 2252 and is configured to use artifact location data to extract a subset or portion of the input image 2202 containing the detected artifacts. This may be done for each detected artifact. For example, the cropping module may be configured to crop or extract a subset of the input image data 2202 defined by bounding box coordinates. The subset of the input image data 2202 corresponding to the detected artifacts may be referred to as the "cropped artifact image". In other embodiments, the cropping module may be a component of the image comparison module 2250 or a standalone module between the image comparison module 2250 and the exceptional defect and anomaly classifier module 2254.

[0388] The exceptional defect and anomaly classifier module 2254 includes an exceptional defect and anomaly classifier model. The classifier model is trained to perform an image classification task. The classifier model may also be a binary classification model configured to receive an input image and assign an exceptional defect label or an anomaly label to the input image.

[0389] The exceptional defect and anomaly classifier model takes image comparison output data 2252 (or cropped artifact images, if generated) as input and generates classification output data 2256. The classification output data 2256 includes the determined class labels. The determined class labels may be linked to artifact identifiers (and also to location data for the artifacts) assigned to the classified artifacts. The classification output data 2256 may also include artifact identifiers and artifact location data for the classified artifacts. In some cases, the classification output data 2256 may include classification output data for multiple detected artifacts (e.g., each artifact from the input image 2202 detected and classified by the golden sample module 2226).

[0390] Classification output data 2256 is provided from the golden sample module 2226 to the anomaly detection module 2240.

[0391] The anomaly detection module 2240 receives OD output data 2218 and classification output data 2256. The OD output data 2218 and classification output data 2256 may each include input image identifier data that identifies the input image 2202 from which the OD output data 2218 and classification output data 2256 were obtained, or they may be linked to or associated with such input image identifier data. The anomaly detection module 2240 may use such input image identifiers to compare the OD output data 2218 and classification output data 2256 corresponding to the same input image 2202.

[0392] The anomaly detection module 2240 is configured to compare the OD output data 2218 with the classification output data 2256 to determine whether an object (defect) detected by the object detection module 2204 corresponds to an exceptional defect determined by the golden sample module 2226. This may include the anomaly detection module 2240 determining whether the location data of the object detection defect and the location data of the exceptional defect satisfy a predetermined condition. For example, the predetermined condition may be that the bounding box coordinates of the exceptional defect are partially or entirely within the bounding box coordinates of the object detection defect (or vice versa). Generally, the predetermined condition is a correlation between the location data of the object detection defect and the exceptional defect such that it can be inferred that they identify the same “item”. If the predetermined condition is met, the anomaly detection module 2240 may generate and assign a “confirmed defect” status or label to the defect. The system may then save the confirmed defect as a “confirmed defect”. In some cases, confirmed defects may be visually labeled as “confirmed” when annotating the input image. Furthermore, determining the confirmed defect status may stop further inspection of the item and initiate downstream processes (for example, by generating a signal and sending it from node device 310 to PLC328).

[0393] If the anomaly detection module 2240 determines that an exceptional defect does not correspond to an object detection defect, the exceptional defect label may be retained. The exceptional defect label data may be used by downstream processes of the system, such as identifying input images containing exceptional defects as training samples to further train the object detector model 2205 to detect exceptional defects, or triggering user interface presentations (e.g., including an annotated image) that require user confirmation of the defect.

[0394] The anomaly detection module 2240 generates anomaly detection output data 2242. The anomaly detection output data 2242 may be an annotated input image. The annotated image may include an input image in which one or more bounding boxes are superimposed, enclosing objects detected by the object detection module 2204 and artifacts detected by the golden sample module 2226. Each bounding box may include an associated label indicating whether the enclosed item is an object detection defect, an exceptional defect, a confirmation defect, or an anomaly. Otherwise, the anomaly detection output data 2242 may include a data structure that stores data defining the detected objects and artifacts (e.g., class label, location data, identifier data, input image identifier data, size data, confidence level data, confirmed status). The data structure may be used to generate an annotated input image, for example, by using an image annotation module to extract the data structure and input image 2202 and generate (and display) the annotated input image. The data structure (or a subset of the data contained therein) may be used for analytical purposes, such as by providing the data structure to the server node 334 for storage in the analytical database 342.

[0395] Referring to Figure 23, an automated visual inspection pipeline 2300 for camshaft visual inspection according to one embodiment is shown.

[0396] The automated visual inspection pipeline 2300 is similar to the automated visual inspection pipeline 2200 and the anomaly detection pipelines 500, 1500, and 1600 described in Figure 22. To avoid repetition, similar content will not be discussed. Therefore, except for the differences discussed below, all content described for pipelines 500, 1500, 1600, and especially pipeline 2200 may also apply to pipeline 2300.

[0397] In Figure 2300, elements and components that have similar functions to those of pipeline 2200 (and pipelines 500, 1500, and 1600) are indicated by the same reference number with the same last two digits.

[0398] Pipeline 2300 begins with inspection image 2302, which is a camshaft. In a variation, inspection image 2302 may be another item or any suitable input image.

[0399] The inspection image 2302 is provided to the object detection module 2304. The object detection module 2304 includes an object detection model with an object suggestion component 2306, a feature map component 2308, an ROI pooling component 2310, a fully connected layer component 2312, a softmax component 2314, and a regressor component 2316.

[0400] The object detection module 2304 generates OD output data 2318. The OD output data 2318 includes the detected defect 2319 (object detection defect 2319). The detected defect 2319 is defined by object data that includes the bounding box (position data) surrounding the object. The OD output data 2318 is provided from the object detection module 2304 to the anomaly detection module 2340.

[0401] The inspection image 2302 is also provided for the golden sample module 2326.

[0402] The inspection image 2302 is provided to the generative model 2330. The generative model 2330 generates the generated golden sample image 2328.

[0403] The inspection image 2302 and the generated golden sample image 2328 are provided to an image comparison module 2350 configured to perform a direct image comparison and generate image comparison output data 2360.

[0404] In one embodiment, a direct image comparison is performed using matrix subtraction or pixel-level grayscale subtraction. The input and output are of the same dimension (identical to the actual input image (e.g., "512*512" from an autoencoder is scaled back to "300*1000")), and only one value is subtracted per pixel. This is in contrast to a feature map approach, which may be performed in feature map analysis 536, etc., in Figure 5, where a downsampled version of the image (e.g., "7*7") is used, and each pixel is compared to a vector of similar size from the same pixel in the feature map of the input image.

[0405] Image comparison output data 2360 includes artifacts 2362, 2364, and 2366 identified by the image comparison process. In one embodiment, image comparison output data 2360 is a grayscale image (e.g., "300*1000") containing the difference between the inspection image 2302 and the generated golden sample image 2328. As previously stated, the difference generally corresponds to "artifacts" (i.e., differences) that are present in the inspection image 2302 but not in the golden sample image 2328.

[0406] Each artifact, 2362, 2364, and 2366, has associated artifact data, including bounding box coordinates that describe the artifact and define the bounding box surrounding the artifact (shown in Figure 23).

[0407] Image comparison output data 2360 is provided to the binarization and shape analysis module 2368.

[0408] The binarization and shape analysis module 2368 performs binarization on the image comparison output data 2360. Binarization involves generating a grayscale image using a static threshold.

[0409] The binarization and shape analysis module 2368 also performs shape analysis on the image comparison output data 2360. The shape analysis includes a combination of morphological operations.

[0410] The output of the binarization and shape analysis module 2368 is provided to the region cropping module 2370. The region cropping module 2370 is configured to crop a subset of image data (i.e., a smaller image) from an image for use in downstream analysis units (e.g., by cropping the bounding box). The region cropping module 2370 may crop the smaller image using the provided artifact position data (e.g., bounding box coordinates).

[0411] The output of the region cropping module 2370 may be a set of small images. In one embodiment, the output may be a batch of "224*224" images.

[0412] The output of the region cropping module 2370 is provided to a classifier model 2372, such as a neural network (e.g., a CNN). In the embodiment shown in Figure 23, the classifier model 2372 is a hybrid CNN binary classifier. In other embodiments, the classifier model 2372 may be a fine-tuned CNN. The hybrid CNN binary classifier may be a combination of convolutional layers (from a pre-trained CNN) and a support vector machine ("SVM") classifier. The SVM classifier may be trained on a small set of images specific to the project. If sufficient training samples are available, the hybrid CNN may be replaced with a fine-tuned CNN.

[0413] The classifier model 2372 takes image data (region-cropped images) as input and is configured to classify the images as exceptional defects 2374 or anomalies 2376. The classification is stored as the artifact's class label.

[0414] In Figure 23, classifier 2372 classified artifact 2362 as an exceptional defect, artifact 2364 as an exceptional defect, and artifact 2366 as an anomaly (as shown in output image 2348, described later).

[0415] The binarization and shape analysis module 2368, the region cropping module 2370, and the classifier model 2372 may also be components of an exceptional defect and anomaly classifier module, such as the exceptional defect and anomaly classifier module 2254 in Figure 22. The classification data 2374 and 2376 are components of the classification output data (e.g., output data 2256) provided to the anomaly detection module 2340.

[0416] The anomaly detection module 2340 receives OD output data 2318 from the object detection module 2304 and classification output data, including classifications 2374 and 2376, from the golden sample module 2326.

[0417] The anomaly detection module 2340 generates an annotated output image 2348, which includes an input image 2302 and a plurality of bounding boxes superimposed on the input image 2302, along with associated labels. In particular, the annotated output image includes an object detection defect 2319, an exceptional defect 2378 (corresponding to artifact 2362), an exceptional defect 2380 (corresponding to artifact 2364 classified as an exceptional defect 2374), and an anomaly 2382 (corresponding to artifact 23666 classified as an anomaly 2376). In some embodiments, the anomaly detection module 2340 may be configured to determine that the object detection defect 2319 and the exceptional defect 2378 correspond to the same defect (for example, by comparing bounding box coordinates or other positional data), generate a single bounding box for the defect (which may correspond to the object detection defect bounding box, the exceptional defect bounding box, or a newly defined bounding box derived from both), and assign the label "confirmed defect" to the defect.

[0418] The above description provides examples of one or more devices, methods, or systems, but it is understood that other devices, methods, or systems may fall within the scope of the claims as interpreted by those skilled in the art.

Claims

1. A method for automated visual inspection of goods, The inspection image of the article is provided as input to an object detection model, which is trained to detect at least one defect type in the input image and generate object position data that identifies the location of the detected object in the inspection image using the object detection model. The inspection image is provided as input to a first convolutional neural network ("CNN"), and the inspection image feature map is generated using the first CNN. The golden sample image of the aforementioned item is provided as input to a second CNN, and a golden sample feature map is generated using the second CNN. The artifact location data, which describes the location of the artifact in the inspection image, is defined, and in order to identify the artifact corresponding to the difference between the inspection image and the golden sample image, the inspection image and the golden sample image are compared. The artifact position data is determined to match the object position data according to predetermined matching criteria. A method for providing this.

2. The method according to claim 1, further comprising confirming the detected object as a defect if the artifact location data matches the object location data.

3. The method according to claim 1, further comprising displaying the detected artifact via a user interface configured to receive input data from the user indicating whether the detected artifact is an exceptional defect or anomaly if the artifact location data does not match the object location data.

4. The system receives input data from the user indicating that the detected artifact is an exceptional defect, The aforementioned inspection image is tagged as an object detection training sample, The method according to claim 3, further comprising:

5. The method according to claim 4, further comprising initiating retraining of the object detection model using the object detection training samples.

6. The method according to claim 1, wherein the golden sample image is a reference image representing a clean image of the article.

7. The method according to claim 1, further comprising generating the golden sample image from the inspection image using a generative machine learning model.

8. Using the object detection model, generate a first defect class label for the detected object, To provide at least a portion of the inspection image, including the detected object, as input to the classification model, Using the classification model, a second defect class label is generated for the detected object. If the first defect class label matches the second defect class label, the first defect class label is verified, The method according to claim 1, further comprising:

9. A computer system for automated visual inspection of goods, The system comprises an input interface for receiving an inspection image of the said article, at least one processor, and a memory communicating with the processor, The at least one processor is The inspection image of the article is provided as input to an object detection model, which is trained to detect at least one defect type in the input image and generate object position data that identifies the location of the detected object in the inspection image using the object detection model. The artifact location data, which describes the location of the artifact in the inspection image, is defined, and in order to identify the artifact corresponding to the difference between the inspection image and the golden sample image, the inspection image and the golden sample image are compared. The artifact position data is determined to match the object position data according to predetermined matching criteria. A system configured to perform the following actions.

10. The system according to claim 9, wherein the at least one processor is further configured to perform the task of confirming the detected object as a defect if the artifact location data matches the object location data.

11. The system according to claim 9, wherein the at least one processor is further configured to display the detected artifact via a user interface configured to receive input data from the user indicating whether the detected artifact is an exceptional defect or anomaly if the artifact location data does not match the object location data.

12. The at least one processor is The system receives input data from the user indicating that the detected artifact is an exceptional defect, The aforementioned inspection image is tagged as an object detection training sample, The system according to claim 11, further configured to perform the following:

13. The system according to claim 12, wherein the at least one processor is further configured to perform the task of initiating retraining of the object detection model using the object detection training samples.

14. The system according to claim 9, wherein the golden sample image is a reference image representing a clean image of the article.

15. The system according to claim 14, wherein the at least one processor is further configured to perform the task of generating the golden sample image from the inspection image using a generative machine learning model.

16. The at least one processor is Using the object detection model, generate a first defect class label for the detected object, To provide at least a portion of the inspection image, including the detected object, as input to the classification model, Using the classification model, a second defect class label is generated for the detected object. If the first defect class label matches the second defect class label, the first defect class label is verified, The system according to claim 9, further configured to perform the following:

17. A method for validating data output from a machine learning model, which runs on one or more processors, To provide image data as input to a first machine learning model having a first set of model parameters and a second machine learning model having a second set of model parameters different from the first set, Using the first machine learning model, the first object, which includes first object attribute data defining the attributes of the first object, is detected in the image data. Using the second machine learning model, the second object, which includes second object attribute data defining the attributes of the second object, is detected in the image data. If the first object attribute data matches the second object attribute data according to predetermined matching criteria, the confirmed status is assigned to the first object. If the first object attribute data does not match the second object attribute data according to the predetermined matching criteria, an unverified status is assigned to or maintained for the first object. A method for providing this.

18. The aforementioned attribute is the object's position. The first object attribute data and the second object attribute data each include position data indicating the respective positions of the first or second object in the image data. The method according to claim 17.

19. The method according to claim 17, wherein the first and second machine learning models each comprise a neural network.

20. The method according to claim 19, wherein the neural network is a convolutional neural network.

21. The method according to claim 1, wherein comparing the inspection image with the golden sample image includes comparing the inspection image feature map of the inspection image with the golden sample feature map of the golden sample image.

22. The first convolutional neural network ("CNN") is used to generate the inspection image feature map, Using the second CNN, generate the aforementioned golden sample image feature map, The method according to claim 21, further comprising:

Images