Systems, methods, and computer devices for artificial intelligence visual inspection using multi-model architectures

Abstract

A system, method, and computer device for automated artificial intelligence visual inspection using a multi-model architecture is provided. The computer device includes a communication interface for receiving image data, a memory for storing the image data, a first neural network model, a second neural network model, and a second neural network model trigger condition, and a processor in communication with the memory. The processor is configured to: perform a first object detection task on the image data using the first neural network model, store the first neural network model output data in the memory, determine whether the first neural network model output data satisfies the second model trigger condition, and perform a second object detection task on the image data using the second neural network model if the first neural network model output data satisfies the second model trigger condition.

Description

[Technical Field]

[0001] The following relates to automated visual inspection for manufacturing quality control, and more particularly to systems and methods for automated visual inspection using artificial intelligence (AI). [Background technology]

[0002] Traditional approaches to automated visual inspection for manufacturing quality control focus on training a single model to perform a variety of different object detection tasks, ranging from recognizing whether a particular object is of a class of object being analyzed to detecting defects or anomalies in a particular object. This approach is very challenging, and incorporating new training data and features into a single model risks performance degradation. This degradation occurs because the model drifts toward newer features, resulting in improved performance on one task while losing its advantage on other tasks when newer data is presented. Summary of the Invention [Problem to be solved by the invention]

[0003] Thus, training large AI models for multiple different purposes to perform multiple different tasks often results in increasingly poor performance, e.g., in terms of weight sharing, network heads, and resource contention.

[0004] Therefore, there is a need for improved systems, methods, and devices for automated visual inspection tasks that overcome at least some of the shortcomings of current systems and methods. Such improved systems, methods, and devices may advantageously perform automated visual inspection tasks related to object detection. [Means for solving the problem]

[0005] A system for automated artificial intelligence (AI) visual inspection using a multi-model architecture is provided. The system includes a camera device for acquiring inspection image data of a target object being inspected and an AI visual inspection device. The AI ​​visual inspection device includes a memory for storing a second model trigger condition for triggering use of a second neural network model, and a processor in communication with the memory. The processor is configured to: execute a first neural network model configured to detect a first object class in the inspection image and generate first neural network model output data including a first list of detected objects; execute a model trigger determination module configured to determine whether the first neural network model output data satisfies the second model trigger condition; execute a second neural network model configured to detect a second object class in the inspection image and generate second neural network model output data including a second list of detected objects when the second model trigger condition is satisfied; and send the neural network model output data, including the first neural network model output data and, if generated, the second neural network model output data, to an operator device via a communication interface. The operator device is configured to display the received neural network model output data.

[0006] In some embodiments, at least one of the first neural network model and the second neural network model is an image segmentation neural network model. In some embodiments, the image segmentation neural network model is an instance segmentation neural network model.

[0007] A system for automated artificial intelligence (AI) visual inspection using a multi-model architecture is provided. The system includes a camera device for acquiring inspection image data of a target object being inspected, and an AI visual inspection device. The AI ​​visual inspection device includes a communications interface for receiving the inspection image data from the camera device, a memory storing a first neural network model configured to detect a first object class in the inspection image data, a second neural network model configured to detect a second object class in the inspection image data, and a second model trigger condition for triggering use of the second neural network model, and a processor in communication with the memory. The processor is configured to: provide the inspection image data as input to the first object detection model; perform a first object detection task using the first neural network model to generate first neural network model output data; store the first neural network model output data in memory as inspection image annotation data; and determine whether the first neural network model output data satisfies the second model trigger condition. If the first neural network model output data satisfies the second model trigger condition, the processor is further configured to perform a second object detection task including providing the inspection image data as input to a second neural network model and generating second neural network output data using the second neural network model, and storing the second neural network model output data in memory as a subset of the inspection image annotation data. The communication interface is configured to send the inspection image data and the inspection image annotation data to an operator device for display. The system further includes an operator device for displaying the inspection image data and the inspection image annotation data as an annotated inspection image.

[0008] The inspection image annotation data may be stored as metadata for the inspection image data.

[0009] The operator device may be configured to receive input data from a user indicating which of the test image annotation data should be displayed, and to display only the indicated test image annotation data in the annotated test image.

[0010] The first neural network model output data may include an object class label of the detected object, the second model trigger condition may include a required object class label, and the processor may determine whether the object class label of the detected object matches the required object class label.

[0011] The first neural network model output data may include object position data of the detected object, the second model trigger condition may include an object position requirement, and the processor may determine whether the object position data of the detected object satisfies the object position requirement.

[0012] The first neural network model output data may include a confidence level of the detected object, the second model trigger condition may include meeting a minimum confidence level, and the processor may determine whether the confidence level of the detected object meets the minimum confidence level.

[0013] The first neural network model output data may include object size data for the detected object, the second model trigger condition may include meeting a minimum object size, and the processor may determine whether the object size data meets the minimum object size.

[0014] The first neural network model output data may include object attribute data describing at least two attributes of the detected object.

[0015] The at least two attributes may include any two or more of an object location, an object class label, an object trust level, and an object size.

[0016] The second model trigger condition may include a requirement for each of at least two attributes of the detected object, and the processor may be further configured to determine whether the object attribute data satisfies the requirement for each of the at least two attributes of the detected object.

[0017] The first neural network output data may include an identifier that identifies the second model trigger condition to be used by the processor.

[0018] The processor may determine, based on the identifier, that a second model trigger condition is to be used.

[0019] When determining that the second model trigger condition is to be used, the processor may retrieve the second model trigger condition from memory using the identifier to determine whether the first neural network model output data satisfies the second model trigger condition.

[0020] The identifier may comprise model identification data that identifies the first neural network model.

[0021] The inspection image data provided to the second neural network model may comprise a subset of the inspection image data that may be determined from the first neural network model output data, and the second object detection task may be performed using the subset of the inspection image data.

[0022] The processor may be further configured to generate, based on the first neural network model output data, a list of neural network models to be executed by the processor, including the second neural network model, when the processor determines that the second model trigger condition is satisfied.

[0023] The processor may sequentially execute each of the neural network models in the list, where executing each of the neural network models includes providing at least a subset of the inspection image data to each of the neural network models and generating neural network model output data using each of the neural network models.

[0024] The processor may be further configured to dynamically update the list to include additional neural network models to be executed, which may be determined by the processor based on neural network output data generated by previously executed neural network models that satisfy model trigger conditions of the additional neural network models stored in memory.

[0025] The list of neural network models to be executed may comprise a list of multiple alternative neural network models to be executed, each corresponding to a single neural network model.

[0026] The operator device may be configured to generate a user interface for receiving input data that sets the second model trigger condition, and the second model trigger condition may be generated by the operator device or the AI ​​visual inspection device in response to the input data.

[0027] In some embodiments, at least one of the first neural network model and the second neural network model is an image segmentation neural network model. In some embodiments, the image segmentation neural network model is an instance segmentation neural network model.

[0028] A computer-implemented method for automated artificial intelligence (AI) visual inspection using a multi-model architecture is provided, the method including: providing inspection image data as input to a first neural network model configured to detect a first object class in the inspection image data; performing a first object detection task using the first neural network model to generate first neural network model output data; storing the first neural network model output data in memory as inspection image annotation data; determining whether the first neural network model output data satisfies a second model trigger condition stored in memory; and if the first neural network model output data satisfies the second model trigger condition, performing a second object detection task using the second neural network model to generate second neural network output data; and storing the second neural network output data in memory as a subset of the inspection image annotation data.

[0029] The method may further include generating an annotated inspection image using the inspection image data and the inspection image annotation data.

[0030] The method may further include displaying the annotated inspection image in a user interface.

[0031] In some embodiments, at least one of the first neural network model and the second neural network model is an image segmentation neural network model. In some embodiments, the image segmentation neural network model is an instance segmentation neural network model.

[0032] A computing device for performing object detection using a multi-model architecture is also provided. The device includes a communications interface for receiving image data, a memory for storing the image data, a first neural network model configured to detect a first object class in the image data, a second neural network model configured to detect a second object class in the image data, and a second neural network model trigger condition, and a processor in communication with the memory. The processor is configured to: perform a first object detection task on the image data using the first neural network model to generate first neural network model output data, store the first neural network model output data in the memory, determine whether the first neural network model output data satisfies the second model trigger condition, and, if the first neural network model output data satisfies the second model trigger condition, perform a second object detection task on the image data using the second neural network model to generate second neural network output data, and store the second neural network model output data in the memory.

[0033] In some embodiments, at least one of the first neural network model and the second neural network model is an image segmentation neural network model. In some embodiments, the image segmentation neural network model is an instance segmentation neural network model.

[0034] Other aspects and features will become apparent to those of ordinary skill in the art upon review of the following description of several exemplary embodiments. [Brief explanation of the drawings]

[0035] The following drawings included herein are intended to illustrate various examples of the articles, methods, and apparatus herein.

[0036] FIG. 1 is a schematic diagram of a system for automated visual inspection according to one embodiment.

[0037] FIG. 2 is a block diagram of a computing device of the present disclosure according to one embodiment.

[0038] FIG. 3 is a block diagram of a computer system for automated visual inspection according to one embodiment.

[0039] FIG. 4 is a block diagram of the multi-model visual inspection module of FIG. 3 according to one embodiment.

[0040] FIG. 5 is a flow diagram of a method for automated visual inspection using the multi-model visual inspection module of FIG. 3 according to one embodiment.

[0041] FIG. 6 is a block diagram of an automated visual inspection system according to one embodiment.

[0042] FIG. 7 is a flow diagram of a method of automated visual inspection using the automated visual inspection system of FIG. 4 according to one embodiment.

[0043] FIG. 8 shows an example display of input and output images of a camshaft, respectively, provided to / by a system for automated visual inspection using a single object detector.

[0044] 9 illustrates an example display of first and second output images of a camshaft and a combined annotated output image of the camshaft that may be generated and used by the systems and methods of the present disclosure, where the combined annotated output image is generated using the first and second output images of the camshaft that are generated using different automated visual inspection models of a multi-model visual inspection system according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0045] Various apparatus or processes are described below to provide examples of each claimed embodiment. Any embodiment described below does not limit any claimed embodiment, and any claimed embodiment may cover a process or apparatus different from the one described below. Claimed embodiments are not limited to an apparatus or process having all of the features of any one apparatus or process described below, or to features common to multiple or all of the apparatuses described below.

[0046] One or more systems described herein may each be implemented in a computer program running on a programmable computer having at least one processor, a data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. As non-limiting examples, the programmable computer may be a programmable logic unit, a mainframe computer, a server, a personal computer, a cloud-based program or system, a laptop, a personal data assistant, a mobile phone, a smartphone, or a tablet device.

[0047] Each program is preferably implemented in a high-level procedural or object-oriented programming and / or scripting language for communicating with a computer system. However, the programs may be implemented in assembly or machine language, if desired. The language may also be a compiled or interpreted language. Each such computer program is preferably stored on a storage medium or device readable by a general-purpose or special-purpose programmable computer, and, when the storage medium or device is read by the computer, configures and operates the computer to perform the procedures described herein.

[0048] The description of an embodiment having several components in communication with each other does not imply that all such components are required. On the contrary, various optional components are described to illustrate the wide range of possible embodiments of the present invention.

[0049] Furthermore, although process steps, method steps, algorithms, etc. may be described (in the disclosure and / or claims) in a sequential order, such processes, methods, and algorithms may be configured to work in different orders. In other words, any sequence or order of steps that may be described does not necessarily require the steps to be performed in that order. Steps of processes described herein may be performed in any practical order. Moreover, some steps may be performed simultaneously.

[0050] Where a single device or article is described herein, it is clear that multiple devices / articles (whether or not they cooperate) may be used in place of the single device / article. Similarly, where multiple devices or articles are described herein (whether or not they cooperate), it is clear that the single device / article may be used in place of the multiple devices or articles.

[0051] The following relates to automated visual inspection for manufacturing quality control, and more specifically, to systems and methods for automated visual inspection using artificial intelligence (AI). The present disclosure provides systems, methods, and devices for artificial intelligence-based image analysis and visual inspection using a multi-model architecture. The multi-model architecture includes multiple machine learning models, such as neural networks. In one embodiment, the neural networks are object detection models. Generally, each neural network is trained to perform a specific task. Model trigger conditions are used to automatically determine whether to trigger the use of other neural networks in the multi-model architecture based on neural network outputs generated by one or more neural networks in the multi-model architecture. Triggering the use of a neural network may include retrieving the image (or a portion thereof) being analyzed from data storage and providing the image to an input layer of the triggered neural network so that the triggered neural network processes the input data and generates a neural network output. The multi-model architecture implemented by the systems and methods of the present disclosure may function as a type of decision tree with neural networks, including the architecture of the models and specific logic between the models that determines which models are triggered by which model outputs.

[0052] As used herein, the term “object detection” is intended to generally refer to computer vision techniques in which objects are detected or identified in digital images. The term “object detection” as used in this disclosure includes, but is not limited to, the specific computer vision technique of “object detection” in which all instances of a known object class are localized and classified in a digital image. For example, the term “object detection” as used herein is intended to include image segmentation techniques in which the presence of objects in a digital image is marked using a pixel mask for each object in the image. One specific example of image segmentation is instance segmentation, in which objects in a digital image are detected and segmented through the localization of specific objects and the association of pixels belonging to them. Instance segmentation involves identifying each object instance for all known objects in a digital image and assigning a label to each pixel in the digital image. Thus, references to a “model,” “object detection model,” “neural network,” “object detection neural network,” etc. are intended to include embodiments in which an instance segmentation model or neural network is used as well as embodiments in which an “object detection” model or neural network is used.

[0053] In industrial and / or commercial situations, various parts need to be analyzed for mechanical fit before delivery to or use by a customer. Similarly, each of the various parts may be affected by many different classes of defects or anomalies. A manufacturer may not be able to sell such defective parts while retaining customer loyalty and / or complying with applicable laws and / or regulations. Such anomalies may not result in many defective parts. Nevertheless, it may be beneficial to a manufacturer to know what defects and / or anomalies occurred in which parts. Such knowledge allows the manufacturer to trace the problem to a specific piece of equipment, process, supply, or precursor. Such knowledge further allows the manufacturer to correct and prevent the defects or anomalies revealed under analysis.

[0054] Detailed analysis of each of the various parts is costly as a function of time. Human workers are generally not as capable as computers or machines of performing detailed memorization tasks over long periods of time without the attendant loss of detail in the short term and the associated loss of job satisfaction in the long term. Thus, it is highly advantageous for manufacturers to use systems for automated visual inspection to analyze parts and detect defects and / or anomalies.

[0055] While it is possible to train a single "big" model to perform each of the various object detection tasks, this approach is flawed and challenging for both the developer and user of the model or system. Even if manufacturers can supply new data to retrain and update the model, such retraining and updating can further drift the model toward a specific task. That is, it can improve on the new task while deteriorating on other tasks. Further concerns with a single "big" model include weight sharing and network topology concerns. When training a single large network for multiple tasks, issues with "task" contention for capacity can arise. This is because the model can optimize the overall loss while negatively impacting the accuracy of a single task. To solve this problem, losses are connected to each task, and the model attempts to optimize all of them simultaneously. This is an area of ​​active research that does not yet have a solution. When there is a compute budget during test time (performing inference at the edge), using a single, smaller model for each task tends to yield significantly higher accuracy. A further drawback of a single "big" model is that regression testing is difficult. It is difficult to know how new data and new tasks will affect previous tasks when they all share weights and gradients, making it difficult to identify what is causing the inconsistency in a large multi-head model.

[0056] Because of the variety of tasks that a system for automated visual inspection must perform (e.g., object identification, defect detection, defect localization), it would be advantageous to have a system that does not degrade in performance on one task to leave room for performance on others. Thus, it would be highly advantageous for a manufacturer to have a system that is composed of several "smaller" models, each performing its own designated task (the results of each may be combined by the system as a single output).

[0057] While this disclosure describes the invention in the context of object defect detection and visual inspection (including manufacturing quality control and visual inspection), the systems, methods, and devices provided herein may have additional and different applications beyond those described herein, whether in the context of object defect detection and visual inspection (e.g., other computer vision applications, such as autonomous vehicles, medical image analysis, and robotics using manipulation). The machine learning models described herein, referred to as object detection models, may, in other embodiments, be other forms of machine learning models configured to perform machine learning tasks other than object detection. For example, a multi-model architecture may include multiple neural networks configured to perform object detection or other image processing tasks. While the input data in such cases may vary, as may the output data, elements of the present disclosure, such as multiple models and trigger conditions, may operate similarly, including data aggregation at the end of the processes disclosed herein.

[0058] As described herein, the present disclosure provides a multi-model architecture including multiple neural networks configured to receive input data and generate at least one output. The neural networks may be feedforward neural networks. The neural network may have multiple processing nodes. The processing nodes may include a multivariate input layer having multiple input nodes, a hidden layer of at least one node, and an output layer having at least one output node. During operation of the neural network, each node in the hidden layer applies an activation / transfer function and weights to any input reaching that node (from the input layer or another layer beyond the hidden layer). The nodes may provide outputs to other nodes (in subsequent hidden or output layers). The neural network may be configured to perform regression analysis to provide continuous outputs or classification analysis to classify data. The neural network may be trained using supervised or unsupervised learning techniques, as described below. According to supervised learning techniques, a training data set is provided at the input layer along with a set of known output values ​​at the output layer. During the training stage, the neural network may process the training data set. The neural network is intended to learn how to provide outputs for new input data by generalizing information learned from training data during the training stage. Training may be influenced by backpropagation error to determine weights for nodes in the hidden layer to minimize error. Once trained (or optionally during training), test or validation data may be provided to the neural network to provide outputs. The neural network may then cross-correlate inputs provided to the input layer to provide at least one output in the output layer.In each embodiment, the output provided by the neural network preferably approximates the desired output for a given input so that the neural network can satisfactorily process the input data.

[0059] 1, an automated visual inspection system 10 according to one embodiment is shown. The system 10 includes an AI visual inspection device 12 that communicates with a camera device 14, an operator device 16, and a programmable logic controller (PLC) device 18 via a network 20.

[0060] AI visual inspection device 12 may be configured to perform an object detection task. AI visual inspection device 12 may include multiple object detection models. Each object detection model may be a model trained to perform a specific object detection task. Object detection involves detecting instances of specific objects belonging to a specific class in input data, such as input data presented to AI visual inspection device 12. Object detection models may include deep learning techniques and machine learning approaches, such as neural networks (e.g., convolutional neural networks or CNNs). While machine learning object detection approaches predefine relevant features of target objects, such definitions are not required in neural networks.

[0061] The AI ​​visual inspection device 12 may also be configured to perform tasks outside the context of object detection. Such tasks may include other forms of machine learning (ML) or artificial intelligence tasks or non-ML tasks.

[0062] The devices 12, 14, 16, and 18 may be server computers, node computing devices (e.g., JETSON computing devices), embedded devices, desktop computers, notebook computers, tablets, PDAs, smartphones, or other computing devices. The devices 12, 14, 16, and 18 may include a connection to a network 20, such as a wired or wireless connection to the Internet. In some cases, the network 20 may include other types of computer or communication networks. The devices 12, 14, 16, and 18 may include one or more of a memory, a secondary storage device, a processor, an input device, a display device, and an output device. The memory may include random access memory (RAM) or a similar type of memory. The memory may also store one or more applications for execution by the processor. The applications may correspond to software modules comprising computer-executable instructions for performing processes for the functions described below. The secondary storage device may include a hard disk drive, a floppy disk drive, a CD drive, a DVD drive, a Blu-ray drive, or other types of non-volatile data storage. The processor may execute applications, computer-readable instructions, or programs. The application, computer readable instructions or program may be stored in memory or secondary storage, or may be received from the Internet or other network 20 .

[0063] An input device may include any device for inputting information into device 12, 14, 16, 18. For example, an input device may be a keyboard, a keypad, a cursor control device, a touch screen, a camera, or a microphone. A display device may include any type of device for presenting visual information. For example, a display device may be a computer monitor, a flat screen display, a projector, or a display panel. An output device may include any type of device for presenting a hard copy of information (e.g., a printer). An output device may include other types of output devices, such as speakers. In some cases, device 12, 14, 16, 18 may include a processor, an application, a software module, a second storage device, a network connection, any two or more of an input device, an output device, and a display device.

[0064] While devices 12, 14, 16, and 18 are described with various components, those skilled in the art will understand that in some cases devices 12, 14, 16, and 18 may include fewer, additional, or different components. Additionally, while aspects of the implementation of devices 12, 14, 16, and 18 may be described as being stored in memory, those skilled in the art will understand that these aspects may be stored on / read from other types of computer program products or computer-readable media, such as secondary storage devices (including hard disks, floppy disks, CDs, or DVDs, carrier waves from the Internet or other networks, or other forms of RAM or ROM). The computer-readable media may include instructions for controlling devices 12, 14, 16, and 18 and / or processors to execute particular methods.

[0065] Devices 12, 14, 16, and 18 may be described as performing particular actions. It is understood that at least any of these devices may perform the actions automatically or in response to interaction by a user of the device. That is, a user of the device may manipulate one or more input devices (e.g., a touchscreen, a mouse, or buttons) to cause the device to perform the described actions. In many cases, this aspect is not described below but is implicitly understood.

[0066] As an example, it is described below that devices 12, 14, 16, 18 may send information to one or more other devices 12, 14, 16, 18. For example, a user using operator device 16 may manipulate one or more inputs (e.g., a mouse and keyboard) to interact with a user interface displayed on a display of device 16. Typically, a device may receive a user interface from network 20 (e.g., in the form of a web page). Alternatively, or in addition, the user interface may be stored locally on the device (e.g., a cache of a web page or a mobile application).

[0067] Device 12, 14, 16, 18 may be configured to receive a plurality of pieces of information from a plurality of devices 12, 14, 16, 18 and / or any of the devices.

[0068] Each device 12, 14, 16, 18 that receives information may store the information in a storage database. The storage may correspond to secondary storage of one or more other devices 12, 14, 16, 18. In general, the storage database may be any suitable storage device, such as a hard disk drive, solid-state drive, memory card, or disk (e.g., CD, DVD, or Blu-ray). The storage database may also be locally connected to the device 12, 14, 16, 18. In some cases, the storage database may be located remotely from the device 12, 14, 16, 18 and accessible to the device 12, 14, 16, 18 over a network or the like. In some cases, the storage database may comprise one or more storage devices located at a network cloud storage provider.

[0069] The AI ​​visual inspection device 12 may be a dedicated device specifically designed to perform object (e.g., defect) detection tasks, object (e.g., defect) classification tasks, golden sample analysis tasks, object (e.g., defect) tracking tasks, and other related data processing tasks using inspection images acquired by the camera device 14.

[0070] The camera device 14 acquires image data. The image data may include a single image or multiple images. Multiple images (frames) may be acquired as video by the camera 14. To image an area of ​​the object to be inspected (which may be referred to as an "inspection object" or "target object"), the camera 14 and the inspection object may move relative to one another. For example, to provide suitable inspection from multiple angles, the object may be rotated and multiple images may be acquired by the camera 14 at different positions. The camera 14 may be configured to acquire multiple frames taken at each position (e.g., when the object rotates relative to the camera 14).

[0071] An inspection object (not shown) may be any physical item for which a user of system 10 wishes to perform a visual inspection. The inspection object may be susceptible to defects developed during a manufacturing or machining process. A defect may be characterized as an unacceptable deviation from a "perfect" or "good" item. An inspection object having a defect is interpreted as defective, unacceptable, or "bad" (NG). System 10 inspects the object and determines whether the object has a defect. The object may be classified by system 10 as defective or non-defective. By identifying the object as defective or non-defective, the inspection object may be treated differently based on the results of the visual inspection. Defective objects may be discarded or removed from further processing. Non-defective objects may proceed to further processing.

[0072] Generally, an inspection object may be any object in which defects are undesirable. Defects in an inspection object may lead to reduced functional performance of the object or a larger object (e.g., a system or device) that includes the inspection object as a component. Defects in an inspection object may reduce the visual appeal of the item. Discovering defective products may be an important step for businesses to prevent the sale and use of defective items and to determine the root causes for the defects and eliminate such causes.

[0073] The inspection object may be an assembled article. The inspection object may be a manufactured article that is susceptible to defects that develop during the manufacturing process. The object may also be an article that derives some value from its appearance, and certain defects may adversely affect its appearance. Defects in the inspection object may develop during the manufacturing of the object itself or some other process (e.g., transfer, testing).

[0074] The test object may be constructed from one or more materials such as metal, steel, plastic, composite, wood, glass, etc.

[0075] The test object may be uniform or non-uniform in size and shape. The object may have a curved outer surface.

[0076] An inspection object may include multiple sections. Object sections may be further divided into object subsections. Object sections (or subsections) may be determined based on the appearance or function of the object. Object sections may be determined to allow for better visual inspection of the object and to better identify unacceptable defective objects.

[0077] Object sections may correspond to different parts of the object with different functions. Different sections may have similar or different dimensions. In some cases, an object may include multiple different section types, each appearing one or more times in the inspection object. Sections may be regular or irregular in shape. Different sections may have different defect specifications (i.e., tolerances for certain defects).

[0078] An inspection object may be susceptible to multiple types or classes of defects detectable using system 10. Examples of defect types may include paint, porosity, dents, scratches, sludge, etc. Defect types may vary from object to object. For example, defect types may be specific to an object based on the manufacturing process or material composition of the object. Defects in an object may be acquired during its manufacture or through subsequent processing of the object.

[0079] The operator device 16 includes a user interface component (or module) (e.g., a human / machine interface). The operator device 16 receives data from the AI ​​visual inspection device 12 over the network 20. The received data may include output data from the camera 14. For example, the output data may include annotated output image data that includes artifact data. The artifact data may include location information (e.g., coordinates, bounding boxes, boundaries of specific instances of objects such as in instance segmentation, centroids) and label information so that artifacts (e.g., defects, anomalies) in the inspection image identified by the AI ​​visual inspection device 12 can be visually identified in the displayed image. Generally, “location information” or “location data” as used herein may include any information or data used to specify or localize the location of an instance of an object in an image and may vary depending on the technique used by the model (e.g., object detection, instance segmentation) to detect objects in an image. Operator device 16 may include automated image annotation software for automatically applying metadata comprising data generated by AI visual inspection device 12 to digital inspection images. Operator device 16 provides output data from AI visual inspection device 12 to a user interface component that generates a user interface screen that displays the annotated output image data. For example, the inspection image may be annotated with metadata comprising defect data generated by the component, such as defect location information (e.g., bounding box coordinates, centroid coordinates), defect size data, and defect class information. Examples of such annotated output images are shown in Figures 8 and 9, described below.

[0080] The user interface component of the operator device 16 may depict one or more user interface elements for receiving input from an operator. For example, the user interface component may provide a "yes / no" or similar binary option for receiving user input data indicating a selection of the option. In certain cases, the user interface may present and highlight a particular object detected by the AI ​​visual inspection device 12 in the annotated output image data and may ask whether the object is anomalous (and may receive corresponding input from the user).

[0081] Depending on the input data received from the user, the annotated output image data (or portions thereof) may be routed differently through system 10. For example, if a user interface component of operator device 16 receives particular input data (e.g., a “no” response to a question about whether a given artifact is anomalous, such as by clicking a user interface element labeled “no”), operator device 16 or AI visual inspection device 12 may be configured to incorporate the new data into its machine learning model. This incorporated data may be recorded as a training sample for a further training data set that can be used to further train one or more artificial intelligence components of AI visual inspection device 12. For example, input data provided via a user interface may be used by operator device 16 or AI visual inspection device 12 to tag or indicate (e.g., by associating metadata) that a particular image generated by system 10 is a training sample for a particular object detection model, which may be part of a multi-model architecture implemented by AI visual inspection device 12 or elsewhere in system 10. Each model in the multi-model architecture may have a model identifier (e.g., model number, name) that can be used for this purpose so that training images can be appropriately tagged for future use in retraining the applicable model.

[0082] PLC device 18 is configured to control the manipulation and physical processing of inspection objects. This may be done by sending and receiving control instructions to and from an article handling unit (not shown) via network 20. Such manipulation and physical processing may include rotating or otherwise driving the inspection object for imaging or for placing and removing the object in the inspection area. An example of an instruction sent by PLC device 18 via network 20 may be "Rotate object 'n' degrees." In some cases, the transmission of such an instruction may depend on information received from AI visual inspection device 12. In other cases, the control instruction may direct the driving or movement of other components of system 10, such as camera 14, conveyor belt, robotic arm, mobile robot, etc., via driving components in communication with PLC 18.

[0083] PLC device 18 may store object detection tolerance data. As an example, the object detection tolerance data may be defect tolerance data (e.g., the range within which system 10 detects defects). The defect tolerance data may include a defect class identifier specific to a particular defect class and one or more tolerance values ​​linked to the defect class identifier. In other embodiments, the defect tolerance data may be stored on another device, such as AI visual inspection device 12. The defect tolerance data may be stored in a defect tolerance database. The defect tolerance data in the defect tolerance database may be referenced using the defect class identifier to facilitate retrieval of the tolerance data values ​​for comparison with data generated by AI visual inspection device 12. Additionally, PLC device 18 may be configured to control the operation of the inspection object. When defects in the inspection object are detected by AI visual inspection device 12, PLC device 18 may compare the detected defects (data related to or describing attributes of the defects) with defect tolerance data stored on PLC device 18 or elsewhere to determine whether the part is defective (e.g., by comparing the defects found in the defect tolerance data).

[0084] For example, in one embodiment, PLC device 18 is configured to receive data indicative of the results of the defect detection process from AI visual inspection device 12 via network 20. For example, if a defect is detected by AI visual inspection device 12, the defect data may be sent to PLC device 18. The defect data describes attributes of the detected defect and may include size data, position data, class label data, confidence level data, etc. PLC device 18 stores defect tolerance data. PLC device 18 analyzes the defect data against the tolerance data and determines whether the inspection object is defective (e.g., “NG”) or within tolerance (e.g., “OK”). PLC device 18 may send a signal indicative of the result of the tolerance analysis to AI visual inspection device 12. If PLC device 18 determines that the defect data is out of tolerance, PLC device 18 may stop inspection of the inspection object and may initiate a process for removing the defective object and installing a new object. PLC device 18 may generate a control signal to stop inspection of the inspection object and may send a control signal to an actuator or other drive component responsible for operating the inspection object.

[0085] If system 10 does not detect any defects in the inspection image, AI visual inspection device 12 sends a signal (e.g., over network 20) ​​to PLC device 18 indicating the result of the object detection process indicating that no defects were found in the image (i.e., "OK"). Upon receiving the "OK" message, PLC device 18 sends a control signal to an actuator or manipulator of (or on) the inspection object to adjust the current inspection position of the inspection object (e.g., rotate the inspection object by "X" degrees). In other cases, control instructions may be sent to other actuation components (e.g., camera actuators) configured to actuate one or more components in response to the received control signal.

[0086] In other embodiments, defect tolerance data may be stored at AI visual inspection device 12, and tolerance analysis may be performed by AI visual inspection device 12. AI visual inspection device 12 may then send a signal to PLC device 18 indicating whether the object is defective. PLC device 18 may then generate a control signal in response to the signal received from AI visual inspection device 12.

[0087] Referring to Figure 2, there is shown a block diagram of one embodiment of a computing device 1000 of system 10 of Figure 1. Computing device 100 may be, for example, any of devices 12, 14, 16, or 18 of Figure 1.

[0088] Computing device 1000 includes several components, such as a processor 1020, which controls the operation of computing device 1000. Communication functions, including data communication, voice communication, or both, may be accomplished through a communications subsystem 1040. Data received by computing device 1000 may be decompressed and decoded by a decoder 1060. Communications subsystem 1040 may receive messages from and send messages to wireless network 1500.

[0089] Wireless network 1500 may be any type of wireless network, including, but not limited to, a data-centric wireless network, a voice-centric wireless network, and a dual-mode network that supports both voice and data communications.

[0090] Computing device 1000 may be a battery-powered device and may include a battery interface 1420 for receiving one or more rechargeable batteries 1440 as shown.

[0091] The processor 1020 also interacts with additional subsystems, such as random access memory (RAM) 1080, flash memory 1110, a display 1120 (e.g., having a touch-sensitive overlay 1140 connected to an electronic controller 1160, which together form a touch-sensitive display 1180), an actuator assembly 1200, one or more optional force sensors 1220, an auxiliary input / output (I / O) subsystem 1240, a data port 1260, a speaker 1280, a microphone 1300, a near-field communication system 1320, and other device subsystems 1340.

[0092] In some embodiments, user interaction with the graphical user interface may be performed through touch-sensitive overlay 1140. Processor 1020 may interact with touch-sensitive overlay 1140 via electronic controller 1160. Information such as text, characters, symbols, images, icons, and other items that may be generated by processor 1020 and displayed or depicted on a computing device may be displayed on touch-sensitive display 1180.

[0093] The processor 1020 may interact with the accelerometer 1360. The accelerometer 1360 may be utilized to detect the direction of gravity or a reaction force resulting from gravity.

[0094] To identify a subscriber for network access according to the present embodiment, computing device 1000 may use a subscriber identity module or removable user identity module (SIM / RUIM) card 1380 inserted into SIM / RUIM interface 1400 for communication with a network (e.g., wireless network 1500). Alternatively, user identity information may be programmed into flash memory 1110 or implemented using other techniques.

[0095] Computing device 1000 also includes operating system 1460 and software components 1480 executed by processor 1020, which may be stored on a persistent data storage device such as flash memory 1110. Additional applications may be loaded onto computing device 1000 through wireless network 1500, auxiliary I / O subsystem 1240, data port 1260, near-field communications subsystem 1320, or any other suitable device subsystem 1340.

[0096] In use, received signals such as text messages, email messages, web page downloads, or other data may be processed by communications subsystem 1040 and input to processor 1020, which then processes the received signals for output to display 1120 or auxiliary I / O subsystem 1240. Subscribers may, for example, create data items such as email messages that may be transmitted over wireless network 1500 via communications subsystem 1040.

[0097] For voice communications, the overall operation of computing device 1000 may be similar. Speaker 1280 may output audio information converted from an electrical signal, and microphone 1300 may convert audio information into an electrical signal for processing.

[0098] Referring to Figure 3, a block diagram of a computing system 300 for automated visual inspection is shown, according to one embodiment. Computer system 300 may be implemented by one or more devices of automated visual inspection system 10 of Figure 1. For example, components of computer system 300 may be implemented by AI visual inspection device 12, operator device 16, and / or PLC device 18 of Figure 1.

[0099] The system 300 includes a processor 302 for executing software models and modules.

[0100] The system 300 further includes a memory 304 for storing data, including output data from the processor 302 .

[0101] The system 300 further includes a communication interface 306 for communicating with other devices, for example, through sending and receiving data over a network connection (eg, network 20 of FIG. 1).

[0102] The system 300 further includes a display 308 for displaying in a human-readable format various data generated by the computer system 300. For example, the display may be configured to display the results of the inspection of the inspection object.

[0103] Processor 302 includes a multi-model visual inspection module 310. Multi-model visual inspection module 310 includes multiple machine learning models configured to perform an object detection task. The multiple machine learning models include a first object detection model 312a, a second object detection model 312b, and a third object detection model 312c. As mentioned above, in other embodiments (e.g., embodiments directed to tasks other than visual inspection), model 312 may be a machine learning model configured to perform a task other than object detection.

[0104] Multi-model visual inspection module 310 processes images of an inspection object received from input module 306 through first model 312a, second model 312b, third model 312c, etc. Those skilled in the art will appreciate that multi-model visual inspection module 310 may include additional models for visual inspection of images of an inspection object. Some or all of the models included in multi-model visual inspection module 310 may be capable of inspecting an image at any given time. Each of the models included in multi-model visual inspection module 310 is trained to perform a particular object detection task unique to that model.

[0105] The memory 304 stores inspection image data 320. The computer system 300 receives the inspection image data 320 via the communication interface 306. The inspection image data 320 may be provided to the computer system 300 by a camera device (e.g., camera 14 of FIG. 1) or other device, such as a remote computing or storage device.

[0106] This input may be received at input module 306, for example, from camera 14 of FIG.

[0107] Through communication between memory 304 and processor 302, inspection image data 320 is provided to first model 312a. As one example, first model 312a may be configured to determine the class of objects or the specific objects, if any, shown in inspection image data 320. As another example, first model 312a may be configured to detect the presence of defects.

[0108] The results of the analysis of the inspection image data 320 by the first model 312a are stored in memory 304 as first model output data 322a. The first model output data 322a may comprise the inspection image data 320 with annotations, such as geometric shapes surrounding areas where objects or defects are recognized, or further labels identifying the objects or defects. In other cases, the first model output data 322a may include only defect data (i.e., data describing any objects identified by the model).

[0109] The model output 322 may be an image or image data. The model output 322 may be annotated with location information (e.g., coordinates, center of gravity / centre location) for the part or defects therein. Additionally, the model output 322 may be annotated with labels for the defect and class assignment (e.g., a defect may be classified as "scratch"). The model output 322 may be annotated with labels for the part and assembly evaluation (e.g., a seal on a part may be identified as properly placed).

[0110] The processor 302 includes a model trigger determination module 316. The model trigger determination module 316 may be located within the multi-model visual inspection module 310. The first model output data 322a is provided to the model trigger determination module 316 in the processor 302. The model trigger determination module 316 uses the first model output data 322a as input to determine which, if any, other models in the processor 302 the inspection image data 320 should be provided to. The model trigger determination module 316 may use artificial intelligence and / or machine learning to make this and other determinations.

[0111] For example, if the first model 312a determines that the inspection image data 320 shows a particular machine part (e.g., by performing object detection to determine a class label corresponding to the part and location information for the part), the model trigger determination module 316 may determine that the inspection image data 320 (or a subset thereof) should be provided to the second model 312b. In contrast, if the first model 312a determines that the inspection image data 320 does not show a particular machine part because the determination is inconclusive or the inspection image data is unclear, the model trigger determination module 316 may determine that the inspection image data 320 should instead be sent to the third model 312c for further processing. Alternatively, if the inspection image data 320 shows a second particular machine part, the inspection image data 320 is provided to the third model 312c for further processing.

[0112] As a further example, the first model 312a may be configured to detect a particular part, area, or region on an object or article. The detected part, area, or region may be prone to the development of a particular type of defect. Therefore, it may be advantageous to determine when such a part, area, or region on an object or article is present (i.e., detected by an object detection model configured to detect the presence of the part, area, or region in the image data) and perform defect detection targeted to the particular part, area, or region. When a part, area, or region is detected by the first model 312a, the inspection image data 320 may be provided to a second model 312b, which may be configured to detect defects in the part, area, or region. The defects that the second model 312b is configured to detect may be specific to the part, area, or region previously detected by the first model 312a (i.e., it may be reasonable to perform defect detection only for the specific defects in the detected part, area, or region). The decision to use the second model 312 b based on the output of the first model 312 a is made according to model trigger conditions 326 stored in the memory 304 .

[0113] As a further example, the first model 312a may be configured to detect the presence of a defect. If the first model 312a detects a defect in the inspection object, the model trigger determination module 316 may determine that the inspection image data 320 should be sent to a second model 312b to further localize the defect and / or to a third model 312c to further classify the defect.

[0114] As a further example, the first model 312a may be configured to detect the presence of a particular component in the inspection image data 320. If the component is present, the model trigger determination module 316 may determine that the inspection image data 320 should be sent to the second model 312b to determine whether the component is properly assembled. For example, if the component is a seal ring assembly, the second model 312b may verify the placement of the seal. The model trigger determination module 316 may then determine that the inspection image data 320 should be further sent to the third model 312c to detect the presence of defects in the inspection image data 320. The assembly detection and defect detection described herein may be performed simultaneously or sequentially in different models 312.

[0115] The above steps may be repeated with respect to providing inspection image data 320 to a second model 312b. The second model output data 322b is stored in memory 304 and is also provided to a model trigger determination module 316 in the processor 302. Depending on the determination of the model trigger determination module 316 regarding the second model output data 322b, the inspection image data 320 may be provided to a third model 312c or additional models.

[0116] Alternatively, after completing analysis by a model 312, the model trigger determination module 316 may send the inspection image data 320 to multiple other models 312 simultaneously or sequentially. For example, after completing analysis by a first model 312a, the model trigger determination module may send the inspection image data 320 to both a second model 312b and a third model 312c, or may send the inspection image data 320 first to the second model 312b and then to the third model 312c. Sending the inspection image data 320 to the third model 312c may be in addition to any further sending of the inspection image data 320 determined by the model trigger determination module 316. Sending the inspection image data 320 to multiple models 312 may be done according to a depth-first approach, a breadth-first approach, or any other approach.

[0117] The determinations made by the model trigger determination module 316 are informed by preset model trigger conditions 326 stored in the memory 304. Such model trigger conditions 326 may be set or modified based on user input. For example, the model trigger conditions 326 may include a condition that if the inspection image data 320 shows a particular machine part as determined by the first model 312a, then the inspection image data 320 should be provided to the second model 312b and / or the third model 312c. The model trigger conditions 326 may include a second model trigger condition and a third model trigger condition. The model trigger conditions 326 correspond to outputs from the models 312 used by the model trigger determination module 316 to determine which model or models 312 should be triggered for subsequent analysis of the inspection image data 320. The model trigger determination module 316 may generate a list of models 312 to be triggered in sequence. In this manner, given model output data 322 may trigger the use of a subsequent given model 312.

[0118] In this way, each of the models 312 that should have an opportunity to analyze the inspection image data 320 according to the model trigger conditions 326 does indeed get such an opportunity. This approach advantageously ensures that each of the models 312 can perform its dedicated analysis without "drifting" from its function by accommodating additional tasks. Furthermore, the multi-model approach may advantageously result in improved efficiency in terms of computing time and resources, because models 312 are triggered for use only when circumstances permit their use (i.e., the system 300 determines that a particular model or models 312 should be used based on the output of other models 312).

[0119] Once analysis by the model 312 is complete, the model output data 322 (e.g., first model output data 322a, second model output data 322b, third model output data 322c), or subsets or parts thereof, are combined by the output image annotator module 314.

[0120] The combination of model output data 322 generated by output image annotator module 314 is stored in memory 304 as annotated output image data 324. The annotated output image data 324 may comprise inspection image data 320 with annotations such as coordinates (e.g., defining bounding boxes) of detected objects such as defects or parts, and / or detected object class labels (e.g., defect type / class, part type / class, part assembly status).

[0121] In some cases, the output image annotator module 314 may use the model output data 322 to generate annotated output image data 324 that is stored in the memory 304.

[0122] Annotated output image data 324 is provided by the model output module 318 that is displayed to the user on the display 308 .

[0123] 4 and 5, a multi-model visual inspection module 310 for automated visual inspection and a method 500 for performing visual inspection using the multi-model visual inspection module 310 are shown, according to one embodiment.

[0124] In the multi-model visual inspection module 310, a first model 312a receives inspection image data 320 and generates first model output data 322a. The first model output data 322a is provided to a model trigger determination module 316 for analysis against model trigger conditions 326. In response to the first model output data 322a, the model trigger determination module 316 may subsequently determine that the inspection image data 320 should be provided to a second model 312b. In response to the second model output data 322b, the model trigger determination module 316 may subsequently determine that the inspection image data 320 should be provided to a third model 312c. Alternatively, the model trigger determination module 316 may subsequently determine that the inspection image data 320 should be provided to the third model 312c immediately after the first model 312a. Such determination is made according to model trigger conditions 326, including a second model trigger condition and a third model trigger condition, stored in memory 304. The second model 312b and the third model 312c are triggered when a second model trigger condition and a third model trigger condition are met, respectively, based on an analysis of the first model output data 322a. Triggering a model may include, for example, providing inspection image data, or a subset thereof, to an input layer of the triggered model to generate an output.

[0125] If the model trigger decision module 316 determines to proceed to the third model 312c, a similar decision regarding proceeding to the fourth model 312d or the fifth model 312e (or both) may be made by the model trigger decision module 316 based on the third model output data 322c.

[0126] The model trigger determination module 316 may be configured to evaluate all model trigger conditions against the received model output data 322, for example, by cycling through each condition to determine whether the condition is met. In other cases, the model trigger determination module 316 may be configured to analyze the received model output data 322 and, from this analysis, determine which subset of model trigger conditions 326 should be evaluated with particular model output data 322 (e.g., by determining from the output that the received output is the output of a particular model). This technique may be applied when, for a given model output data 322, only a particular subset of models 312 may (or may not) be triggered, and only the model trigger conditions for these potentially triggered models should be evaluated. For example, if the second model 312b is the only model that can be (or cannot be) triggered by the output data 322a of the first model 312a, then upon determining that the received output is output data 322a from the first model 312a, the model trigger determination module 316 may be configured to evaluate the output data 322a using only the second model trigger conditions (and not bother evaluating it using the unnecessary and inefficient third model trigger conditions).

[0127] The model trigger determination module 316 may be configured to maintain a list or other data structure that indicates which models 312 should be triggered based on the analysis performed by the model trigger determination module 316. The list of triggered models 312 (which may include between 0 and N models if the system 300 has N models) may be used to sequentially trigger the models 312 to perform their respective analyses.

[0128] In one embodiment, the multi-model visual inspection module 310 may include a single model trigger decision module 316 in communication with each of the one or more models 312. The output data of each model 312 is provided for analysis by the single model trigger decision module 316.

[0129] In other embodiments, the model trigger decision module 316 may include multiple model trigger decision modules, where a model trigger decision module may be inserted within a pair of models (not shown) (e.g., between the first model 312a and the second model 312b, or between the first model 312a and the third model 312c) to locally control where the inspection image data 320 is further sent. Such a model trigger decision module may be associated with each subsequent model 312 (e.g., 312b and 312c) instead of being inserted between the previous model 312 (e.g., 312a) and the subsequent model 312 (e.g., 312b or 312c).

[0130] In other embodiments, each model 312 may include an internal model trigger decision module (not shown) that makes the same decision regarding further delivery of inspection image data 320. In one such embodiment, the model trigger decision module is configured to determine satisfaction of the model trigger condition 326 based on an analysis of the model's own output data.

[0131] In other embodiments, each model-triggered decision module may be provided according to any of the above embodiments, i.e., some may be inserted within a pair of models, some may be associated with subsequent models, and some may be contained within a model, so that no such status describes all model-triggered decision modules.

[0132] In each of the above configurations of model trigger decision module 316, there may be only a single model trigger decision module 316 in the multi-model visual inspection module 310, virtually represented within or between each of the models 312 as previously described.

[0133] After the analysis by the model 312 is completed, the model trigger determination module 316 may send the inspection image data 320 to multiple other models 312 simultaneously or sequentially. For example, after the analysis by the first model 312a is completed and the first model output data 322a is generated, the model trigger determination module may send the inspection image data 320 to both the second model 312b and the third model 312c, or may send the inspection image data 320 first to the second model 312b and then to the third model 312c. Sending the inspection image data 320 to the third model 312c may be in addition to any further sending of the inspection image data 320 determined by the model trigger determination module 316. Sending the inspection image data 320 to multiple models 312 may be performed according to a depth-first approach, a breadth-first approach, or any other approach.

[0134] In some cases, the model trigger determination module 316 is configured to coordinate the sequence of possible model 312 operations across multiple decisions based on different model output data 322. For example, the model trigger determination module 316 may maintain a list or other data structure indicating which models 312 are to be triggered, such as by initiating the provision of inspection image data 320 (or a subset thereof) to each model 312. This list of triggered models (or, more precisely, models to be triggered) 312 may be dynamically updated as analysis by the models 312 continues. For example, based on analysis of the first model output data 322a, the model trigger determination module 316 may determine that a first list of models 312 should be triggered based on the output. This first list may include multiple models 312. The model trigger determination module 316 initiates the provision of inspection image data 320 (or a subset thereof) to the first list models in the first list of models 312. The first list models 312 analyze the input inspection image data and generate their own output data 322, which is provided to the model trigger determination module 316. The model trigger determination module 316 may then generate a second list of models 312 to be triggered based on the outputs 322 of the first list models 312 in the first list of models 312. The model trigger determination module 316 may then dynamically update the list of models 312 to be triggered to include the second list of models 312 (in addition to the previously determined first list of models 312). In this manner, the model trigger determination module 316 can manage new model trigger determinations as models 312 are triggered and generate new model output data 322 to be analyzed.

[0135] With particular reference to Figure 4, a block diagram of the multi-model visual inspection module of Figure 3 is shown. In one embodiment, multi-model visual inspection module 310 may cause second model 312b and / or third model 312c to send inspection image data 320 after first model 312a outputs first model output data 322a. Similarly, multi-model visual inspection module 310 may cause fourth model 312d and / or fifth model 312e to send inspection image data 320 after third model 312c outputs third model output data 322c. The determination by multi-model visual inspection module 310 is based on whether model trigger conditions 326 stored in memory 304 are satisfied.

[0136] With particular reference to Figure 5, a method 500 for performing automated visual inspection is shown, according to one embodiment. Method 500 may be implemented by computer system 300 of Figure 3. Method 500 may be directed to automated visual inspection of objects for object detection, or may be further used in other contexts.

[0137] At 502, the system 300 of FIG. 3 receives inspection image data 320, such as from the camera 14 of FIG.

[0138] At 504, the multi-model visual inspection module 310 sends inspection image data 320 to a model 312, such as a first model 312a.

[0139] At 506, the multi-model visual inspection module 310 stores the model output data 322 generated by the object detection models 312 (e.g., the first model output data 322a generated by the first model 312a). The model output data 322 is stored in the memory 304.

[0140] At 508, the model trigger decision module 316 determines whether to send the inspection image data 320 to a subsequent model 312, such as the second model 312b (i.e., whether one or more other models 312 should be triggered). This decision is made based on the model output data 322 (e.g., the first model output data 322a) from 506 and in accordance with the model trigger conditions 326.

[0141] If the model trigger decision module 316 determines 508 as "yes," steps 504-506 are repeated for subsequent models 312, such as the second model 312b.

[0142] If the model trigger determination module 316 determines 'no' at 508, the method 500 instead proceeds to 510.

[0143] At 510, all of the model output data 322 (e.g., the first model output data 322a and the second model output data 322b) are integrated by the output image annotator module 314 as a single annotated output image data 324 that is stored in the memory 304.

[0144] At 512, the model output module 318 sends the annotated output image data 324 to the display 308 for display to the user. This may include depicting the annotated output image in a graphical user interface. In some cases, the user interface may be implemented on a user device, such as operator device 16 of FIG. 1, and the annotated output image may be sent to the user device over a network connection (e.g., network 20 of FIG. 1), etc.

[0145] 6 and 7, an embodiment 600 of the multi-model visual inspection module 310 for automated visual inspection and a method 700 of performing visual inspection using the multi-model visual inspection module 600 of FIG. 6 are shown.

[0146] The module 600 comprises a first object detection model 602, a second object detection model 604, and a third object detection model 606. Although the models 602, 604, 606 are described as having a particular order or sequence, it is understood that the order in which the models are presented (and triggered) may be varied in other embodiments.

[0147] Module 600 includes a first detection model 602. First detection model 602 is a defect detection model configured to detect multiple classes of defects in an input image. The defect classes include scratches, porosity, and dents. In other embodiments, first detection model 602 may include fewer or additional defect classes.

[0148] At 702, the first detection model 602 performs defect detection in the inspection image data 320, as described above. The output of the first detection model 602, such as first model output data 322a, is stored in the memory 304. The output of the first detection model 602 may comprise the inspection image data 320 with annotations, such as geometric shapes surrounding areas where defects are recognized or labels identifying defect types / classes. In other cases, the output of the first detection model 602 may include only defect data (i.e., data describing any defects identified by the model).

[0149] The module 600 includes a second detection model 604. The second detection model 604 is a part section (or "section") model configured to detect multiple classes of part sections in the input image. The part section classes include a VTC class, a journal class, a lobe class, and a censoring class. The second detection model 604 detects and localizes part sections in the input image, such as by generating bounding boxes around the part sections and part section class labels. The detected part sections may also be interpreted as "regions of interest" (and singular "region of interest" or "ROI"). The second detection model 604 identifies part sections present in the input image (which, in the case of a sequence of images, is the current image).

[0150] At 704, the second detection model 604 performs detection of individual parts and part sections in the inspection image data 320, as described above. The output of the second detection model 604, such as second model output data 322b, is stored in memory 304. The output of the second detection model 604 may comprise inspection image data 320 with annotations, such as geometric shapes enclosing areas where parts or part sections are recognized, or labels identifying the parts or part sections (object classes). In other cases, the output of the second detection model 604 may include only part and / or part section data (i.e., data describing any parts and / or part sections identified by the model).

[0151] The module 600 includes a third detection model 606. The third detection model 606 is an assembly detection model configured to detect multiple classes of assembly features in the input image. The assembly classes include a seal ring class and an oil hole class. In this manner, the third detection model 606 detects and localizes assembly features in the input image, such as by generating bounding boxes surrounding the assembly features, assembly feature class labels, and other detected object data. In essence, the third detection model 606 determines whether a given assembly feature corresponding to an assembly feature class is present in the image.

[0152] At 706, the third detection model 606 performs detection of appropriate assemblies in the inspection image data 320, as described above. The output of the third detection model 606, such as third model output data 322c, is stored in memory 304. The output of the third detection model 606 may comprise the inspection image data 320 with annotations, such as geometric shapes enclosing areas where assemblies are recognized, or labels identifying the assemblies (object classes). In other cases, the output of the third detection model 606 may include only assembly data (i.e., data describing any assemblies identified by the model).

[0153] An example of the use of module 600 and method 700 is described below. Generally, first model 602 searches for different types of defects in an inspection image of a camshaft. If defects are found, the image is passed to second detection model 604 for detecting part sections. The decision to pass the image to the second detection model is based on the first model output data satisfying a second model trigger condition. Second detection model 604 locates and identifies key sections of the camshaft on the image. Module 600 checks whether there are defects within the detected part sections. This involves comparing object location data for detected objects (defects, ROI sections). Furthermore, if a particular class of part section is detected in the image (i.e., output data), the image is passed to third detection model 606 to determine (confirm) that a particular assembly feature is within the detected part section. The decision to pass the image to the third detection model is based on the second model output data satisfying a third model trigger condition.

[0154] It will be appreciated that before the inspection image data 320 is sent to each of the models 602, 604, and 606 of module 600, the model trigger determination module 316 may determine whether analysis by a particular model 602, 604, 606 is appropriate in light of the inspection image data 320 and any available output data 322 provided by models that have already analyzed the inspection image data 320. Such determination is made in accordance with model trigger conditions 326. Furthermore, in some cases, the output data 322 generated by a model may be saved and used for analysis using output data 322 from a different model (e.g., comparing two output data). For example, object data, such as bounding box coordinates and defect class labels, of a defect detected by the first model 602 may be saved and later compared by module 600 with object data of the part section detected by the second model 604. If a comparison of the bounding box coordinates determines that a defect of a particular class is present in a part section of a particular class, module 600 may determine the defect as unacceptable and initiate corresponding downstream processes. If the defect is of an acceptable class or if the defect is outside the part section, module 600 may tag the defect as acceptable and / or ignore the defect. Such an approach is particularly advantageous when different part sections or regions of interest have different defect tolerances (e.g., certain types of defects are acceptable / unacceptable in a particular region of interest, whereas defects below a threshold size are acceptable). It is understood that the above concepts may be applied to obtain other relationships between “objects” detected using different models to provide improved capabilities to the visual inspection system. In other words, output data 322 from different models may be analyzed or compared by module 600 to determine subsequent actions, which may include triggering other models. Such comparisons may be embodied in model trigger conditions for the triggered models.For example, the output data 322 of the first model 602 and the second model 604 may be used in determining whether the third model 606 should be triggered.

[0155] It is further understood that each of models 602, 604, and 606 may include additional models that perform further analyses or sub-analyses. For example, first detection model 602 may include several "smaller" models (each of which may receive inspection image data 320 to perform a particular analysis or sub-analysis transmitted by first detection model 602).

[0156] The determination of which "smaller" model (if any) of the models 602, 604, 606 the inspection image data 320 should be sent to is made by the model trigger decision module 316 according to the model trigger conditions 326. As previously mentioned, each model 602, 604, 606, or any sub-model, may include additional model trigger decision modules within or between them. If such additional model trigger decision modules exist within or between them, such additional model trigger decision modules may be virtual modules representing the single model trigger decision module 316 in the module 600.

[0157] 8, there is shown an example display of an image 800 of a camshaft used or generated by a visual inspection system that employs a traditional approach to object detection by having a single object detector perform multiple object detection tasks. Image 800 includes an input inspection image 802 of a camshaft and an annotated output image 804 of the camshaft after visual inspection using the single object detector.

[0158] Image 802 is an inspection image of a camshaft. Inspection image 802 is provided as input to a single object detector configured to perform defect detection, part section detection, and assembly feature detection. The single object detector generates an annotated output image 804. The annotated output image 804 includes inspection image 802 and various information about defects, part sections, and assembly features detected by the single object detector overlaid on the inspection image. In particular, image 804 includes bounding boxes 806a-806g that identify detected objects. Box 806a is output from section detection showing the ROI for the VTC. Box 806b is output from section detection locating the thrust section. Box 806c is output from defect detection showing identified defects. Boxes 806d-806g are visual representations of where defects and objects have been filtered out (e.g., those outside a specific area are ignored).

[0159] 9, an example display of an annotated output image 900 of a camshaft is shown, according to one embodiment, generated by the visual inspection system of the present disclosure. The system used to generate image 900 may offer improvements over the single detector system used in image 802 of FIG.

[0160] Image 900 may be generated by computer system 300 of Figure 3 or AI visual inspection device 12 of Figure 1. Image 900 includes a first annotated output image 902, a second annotated output image 904, and a combined annotated output image 908.

[0161] Image 902 is an example of an image generated using a part section detection model, such as part section detection model 604 of Figure 6. Image 902 is one embodiment of model output 322. Image 902 includes annotations 906a that include bounding boxes indicating locations in image 902 where particular classes of part sections were detected and labels corresponding to the classes of part sections.

[0162] Image 904 is an example of an image generated using a defect detection model, such as defect detection model 602 of Figure 6. Image 904 is one embodiment of model output 322. Image 904 includes further annotations 906b-906f, including bounding boxes indicating the locations in image 904 where particular defects are found and labels indicating the classes of the detected defects.

[0163] The images 902 and 904 are combined in the output image annotator module 314 to generate the annotated output image data 324 .

[0164] Image 908 is one embodiment of annotated output image data 324. The image includes annotations generated from part sections and defect detection models (and present in output images 902, 904 of these models). Image 908 includes annotations 906a-906f (and present in images 902, 904) corresponding to objects detected by the part sections and defect detection models. The annotations include bounding boxes indicating the locations in image 908 where particular detected objects are identified and labels indicating the class of the objects.

[0165] In other embodiments, image 908 may include only a subset of the annotations. Which subset of annotations to use or display may be determined automatically by the system (e.g., by image annotator module 314 or other software logic) or may be based on input provided by a user at a user device. For example, which objects to retain or display may be determined based on meeting a confidence threshold, satisfying a particular region of interest (ROI) filter, having a particular object class (e.g., removing unnecessary classes / objects found), etc.

[0166] The features referred to above may include defects, assemblies, and either the presence or absence of particular parts or sections on or within the inspection object.

[0167] The combined annotated output image 908 may be sent by the model output module 318 to the display 308 for display to a user. The combined annotated output image 908 may be stored in the memory 304. In some cases, the combined annotated output image 908 may be sent to another device (not shown) for storage, such as cloud storage in a cloud device, and / or analysis (e.g., to an analytics server).

[0168] Although embodiments of the invention have been described in which, when inspection image data 320 is sent to multiple models 312, each model 312 receives the inspection image data 320 sequentially, related systems, methods, and devices may be configured such that the models 312 receive the inspection image data 320 substantially and / or completely in parallel / concurrently / simultaneously.

[0169] Although the foregoing description provides one or more example devices, methods, or systems, it will be understood that other devices, methods, or systems may be within the scope of the claims as interpreted by one of ordinary skill in the art.

Claims

1. 1. A system for automated artificial intelligence (AI) visual inspection using a multi-model architecture, comprising: a camera device for acquiring inspection image data of a target object being inspected; a memory for storing a second model trigger condition for triggering use of the second neural network model; executing a first neural network model configured to detect a first object class in the inspection image and generate first neural network model output data including a first list of detected objects; executing a model trigger determination module configured to determine whether the first neural network model output data satisfies the second model trigger condition; executing the second neural network model configured to detect a second object class in the inspection image and generate second neural network model output data including a second list of detected objects when the second model trigger condition is satisfied; sending neural network model output data including the first neural network model output data and, if generated, the second neural network model output data to an operator device via a communications interface; a processor in communication with the memory configured to execute An AI visual inspection device comprising: Equipped with The system wherein the operator device is configured to display the received neural network model output data.

2. 2. The system of claim 1, wherein the first neural network model output data includes an object class label of a detected object, the second model trigger condition includes a requested object class label, and the processor determines whether the object class label of the detected object matches the requested object class label.

3. 2. The system of claim 1, wherein the first neural network model output data includes object position data of a detected object, and the second model trigger condition includes an object position requirement, and the processor determines whether the object position data of the detected object satisfies the object position requirement.

4. 2. The system of claim 1, wherein the first neural network model output data includes a confidence level of the detected object, and the second model trigger condition includes meeting a minimum confidence level, and the processor determines whether the confidence level of the detected object meets the minimum confidence level.

5. 2. The system of claim 1, wherein the first neural network model output data includes object size data for the detected object, the second model trigger condition includes meeting a minimum object size, and the processor determines whether the object size data meets the minimum object size.

6. the first neural network model output data includes object attribute data describing at least two attributes of the detected object; the at least two attributes include any two or more of an object location, an object class label, an object trust level, and an object size; The system of claim 1 .

7. 7. The system of claim 6, wherein the second model trigger condition includes a requirement for each of the at least two attributes of the detected object, and the processor is further configured to determine whether the object attribute data satisfies the requirement for each of the at least two attributes of the detected object.

8. the first neural network model output data includes an identifier that identifies the second model trigger condition for use by the processor; the identifier comprises model identification data that identifies the first neural network model; the processor determines, based on the identifier, that the second model trigger condition is to be used. The system of claim 1 .

9. 9. The system of claim 8, wherein upon determining that the second model trigger condition is to be used, the processor retrieves the second model trigger condition from the memory using the identifier to determine whether the first neural network model output data satisfies the second model trigger condition.

10. 2. The system of claim 1, wherein the inspection images provided to the second neural network model comprise a subset of the inspection images determined from the first neural network model output data, and a second object detection task is performed using the subset of the inspection images.

11. 2. The system of claim 1, wherein the processor is further configured to generate, based on the first neural network model output data, a list of neural network models to be executed by the processor, the list including the second neural network model, when the processor determines that the second model trigger condition is satisfied.

12. 12. The system of claim 11, wherein the processor sequentially executes each of the neural network models in the list, and executing each of the neural network models includes providing at least a subset of the inspection image to each of the neural network models and generating neural network model output data using each of the neural network models.

13. 12. The system of claim 11, wherein the processor is further configured to dynamically update the list to include additional neural network models to be executed, determined by the processor based on neural network output data generated by previously executed neural network models that satisfy model trigger conditions of the additional neural network models stored in the memory.

14. 12. The system of claim 11, wherein the list of neural network models to be executed comprises a list of multiple distinct neural network models to be executed, each corresponding to a single neural network model.

15. 2. The system of claim 1, wherein the operator device is configured to generate a user interface for receiving input data that sets the second model trigger condition, and the second model trigger condition is generated by the operator device or the AI ​​visual inspection device in response to the input data.

16. The system of claim 1 , wherein at least one of the first neural network model and the second neural network model is an image segmentation neural network model.

17. The system of claim 16 , wherein the image segmentation neural network model is an instance segmentation neural network model.

18. 1. A computer-implemented method for automated artificial intelligence (AI) visual inspection using a multi-model architecture, comprising: providing the inspection image data as input to a first neural network model configured to detect a first object class in the inspection image data; performing a first object detection task using the first neural network model, the first object detection task including generating first neural network model output data; storing the first neural network model output data in a memory as inspection image annotation data; determining whether the first neural network model output data satisfies a second model trigger condition stored in the memory; If the first neural network model output data satisfies the second model trigger condition, providing the inspection image data as input to a second neural network model configured to detect a second object class in the inspection image data; performing a second object detection task using the second neural network model, the second object detection task including generating second neural network output data; storing the second neural network output data in the memory as a subset of the inspection image annotation data; How to do it. generating an annotated inspection image using the inspection image data and the inspection image annotation data; displaying the annotated inspection image in a user interface; and The method of claim 18 further comprising:

20. The method of claim 18, wherein at least one of the first neural network model and the second neural network model is an instance segmentation neural network model.

Images