Systems, methods, and computer devices for automated visual inspection using adaptive region of interest segmentation

Abstract

A computer system, method, and device for visual inspection using adaptive ROI segmentation are provided. The system includes a camera for acquiring an inspection image of a target article and an AI visual inspection computing device for detecting defects or anomalies in the target article. The device includes a communication interface for receiving the inspection image, an adaptive ROI segmentation module for processing the inspection image to generate an inspection image in which regions of non-interest are masked, an image analysis module for receiving the masked inspection image and analyzing the masked inspection image with an image analysis model to generate output data indicative of the presence of detected defects or anomalies, where analysis of the masked inspection image is limited to unmasked regions of interest ("ROIs"), and an output interface for displaying the output data.

Description

[Technical Field]

[0001] The following relates generally to machine learning-based visual inspection, and more particularly to visual inspection using adaptive region of interest ("ROI") segmentation. [Background technology]

[0002] Image analysis, object detection, and other procedures often require significant computational resources to thoroughly analyze each portion of an input image, which can become prohibitive as a function of cost and time if not all input images are potentially useful or may reveal valuable information.

[0003] Similarly, with the development of object detection and analysis, false positives can occur when regions of an input image and / or its background may contain features or elements that are similar, but not identical, to a desired object, feature, etc. If these distinctions are not apparent to computer systems and devices performing such object detection and analysis, additional computational resources may be wasted not only on detecting such false positives, but also on downstream operations caused by the false positive detections. This can be particularly problematic in visual inspection tasks where the time to inspect an object is limited, such as in applications such as manufacturing quality control.

[0004] Thus, there is a demonstrated need for a system, method, and apparatus that can mask or block areas of an input image that are not of interest to the computational task being performed. Summary of the Invention [Means for solving the problem]

[0005] A system for visual inspection of a target article using adaptive region of interest ("ROI") segmentation is provided. The system includes a camera and an AI visual inspection computing device for detecting defects or anomalies in the target article. The camera acquires an inspection image of the target article. The AI ​​visual inspection computing device includes a communications interface for receiving the inspection image acquired by the camera, an adaptive ROI segmentation module for processing the inspection image with an ROI segmentation model to generate a masked inspection image in which regions of non-interest ("nROIs") are masked, and an image analysis module for receiving the masked inspection image and analyzing the masked inspection image with an image analysis model to generate output data indicative of the presence of defects or anomalies detected by the image analysis model, wherein analysis of the masked inspection image is limited to the unmasked ROIs. The AI ​​visual inspection computing device further includes an output interface for displaying the output data.

[0006] The image analysis model may include an object detection model trained to detect at least one defect class in a mask inspection image.

[0007] The image analysis model may include a golden sample analysis module configured to compare the mask inspection image to a golden sample image of the target article.

[0008] The image analysis model may include an object detection model and a golden sample analysis module.

[0009] The system may include a comparison module for comparing the object detection output data generated by the object detection model with the golden sample output data generated by the golden sample analysis module.

[0010] The golden sample module may include a generative model for generating a golden sample image from an inspection image.

[0011] The output data may include the defect type and defect location for each defect detected by the object detection model.

[0012] The adaptive ROI segmentation model may be trained to identify and mask non-uniform areas of the inspection image.

[0013] The non-uniform areas may include any one or more of improperly illuminated areas in the inspection image, user-defined non-uniform areas, components of the target article that vary between different target articles of the same class, and irregularly textured areas of the target article.

[0014] The output data may classify the target item as either defective or non-defective.

[0015] A method for visually inspecting a target article using adaptive region of interest ("ROI") segmentation is provided, the method including acquiring an inspection image of the target article, processing the inspection image by masking nROIs in the inspection image with an adaptive ROI segmentation model, analyzing the masked inspection image with an image analysis model to detect defects or anomalies in the target article, generating output data based on an output of the image analysis model, the output data indicating the presence of the detected defects or anomalies, and displaying the output data on a user device.

[0016] The method may include comparing the object detection output data generated by the object detection model to the golden sample output data generated by the golden sample analysis module.

[0017] The output data may include the defect type and defect location for each defect detected by the object detection model.

[0018] The adaptive ROI segmentation model may be trained to identify and mask non-uniform areas of the inspection image.

[0019] The non-uniform areas may include one or more of improperly illuminated areas in the inspection image, user-defined non-uniform areas, components of the target article that vary between different target articles of the same class, and irregularly textured areas of the target article.

[0020] An AI visual inspection computing device for detecting objects in an inspection image using adaptive region of interest ("ROI") segmentation is provided, including a communications interface for receiving an inspection image acquired by a camera, an adaptive ROI segmentation module for processing the inspection image with an ROI segmentation model to generate a masked inspection image in which regions of non-interest ("nROIs") are masked, an image analysis module for receiving the masked inspection image and analyzing the masked inspection image with an image analysis model to generate output data indicative of the presence of objects detected by the image analysis model, where analysis of the masked inspection image is limited to unmasked regions of interest ("ROIs"), and an output interface for displaying the output data.

[0021] The apparatus may include a comparison module for comparing the object detection output data generated by the object detection model with the golden sample output data generated by the golden sample analysis module.

[0022] The output data may include the defect type and defect location for each defect detected by the object detection model.

[0023] The adaptive ROI segmentation model may be trained to identify and mask non-uniform areas of the inspection image.

[0024] The non-uniform areas may include one or more of improperly illuminated areas in the inspection image, user-defined non-uniform areas, components of the target article that vary between different target articles of the same class, and irregularly textured areas of the target article.

[0025] Other aspects and features will become apparent to those of ordinary skill in the art upon review of the following description of several exemplary embodiments. [Brief explanation of the drawings]

[0026] The drawings included herein are intended to illustrate various examples of the articles, methods, and apparatuses herein.

[0027] [Figure 1] FIG. 1 is a schematic diagram of a computer system for adaptive ROI segmentation, according to one embodiment.

[0028] [Figure 2] FIG. 1 is a block diagram of a computing device of the present disclosure, according to one embodiment.

[0029] [Figure 3] FIG. 1 is a block diagram of a visual inspection system including adaptive ROI segmentation, according to one embodiment.

[0030] [Figure 4] FIG. 1 is a block diagram of a computer system for visual inspection using adaptive ROI segmentation, according to one embodiment.

[0031] [Figure 5A] FIG. 1 is a block diagram of a defect detection pipeline for a visual inspection system including adaptive ROI segmentation, according to one embodiment.

[0032] [Figure 5B]FIG. 5B is a block diagram of a training pipeline for training an adaptive ROI segmentation module for use in the defect detection pipeline of FIG. 5A according to one embodiment.

[0033] [Figure 6] 1A-1C show example input and output images and a schematic diagram of an adaptive ROI segmentation device for receiving the input image and generating the output image, according to one embodiment;

[0034] [Figure 7] 1A-1C are diagrams of sets of image pairs including input and output images of a camshaft that are each provided to and generated by a system for ROI segmentation, according to one embodiment.

[0035] [Figure 8] FIG. 1 is a block diagram of a visual inspection processing pipeline for golden sample image analysis with adaptive ROI segmentation, according to one embodiment.

[0036] [Figure 9] FIG. 1 is a flow diagram of a visual inspection method using adaptive ROI segmentation in the context of golden sample image analysis, according to one embodiment.

[0037] [Figure 10] FIG. 1 is a block diagram of a visual inspection pipeline for golden sample image analysis with adaptive ROI segmentation, according to one embodiment.

[0038] [Figure 11] FIG. 1 is a flow diagram of a visual inspection method using adaptive ROI segmentation in the context of golden image analysis, according to one embodiment.

[0039] [Figure 12]FIG. 1 is a block diagram of a visual inspection processing pipeline for golden sample image analysis with adaptive ROI segmentation, according to one embodiment.

[0040] [Figure 13] FIG. 1 is a flow diagram of a visual inspection method using adaptive ROI segmentation in the context of golden sample image analysis, according to one embodiment.

[0041] [Figure 14] FIG. 1 is a block diagram of a visual inspection processing pipeline for object detection image analysis with adaptive ROI segmentation, according to one embodiment.

[0042] [Figure 15] FIG. 15 is a block diagram of the object detection model of FIG. 14 according to one embodiment.

[0043] [Figure 16] FIG. 1 is a flow diagram of a visual inspection method using adaptive ROI segmentation in the context of object detection image analysis, according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0044] Various apparatus or processes are described below to illustrate the claimed embodiments. The embodiments described below do not limit the claimed embodiments, and the claimed embodiments may cover processes or apparatuses different from those described below. The claimed embodiments are not limited to apparatus or processes having all the features of any one apparatus or process described below, nor are they limited to features common to multiple or all of the apparatuses described below.

[0045] One or more systems described herein may be implemented in a computer program executing on a programmable computer, each of which includes at least one processor, a data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. For example, but not limited to, the programmable computer may be a programmable logic unit, a mainframe computer, a server, a personal computer, a cloud-based program or system, a laptop, a personal data assistant, a mobile phone, a smartphone, or a tablet device.

[0046] Each program is preferably implemented in a high-level procedural or object-oriented programming, and / or scripting language for communicating with a computer system. However, the programs can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Each such computer program is preferably stored on a general-purpose or special-purpose programmable computer-readable storage medium or device for configuring and functioning the computer when the storage medium or device is read by the computer to perform the procedures described herein.

[0047] A description of an embodiment having several components in communication with each other does not imply that all such components are required, but rather various optional components are described to illustrate the wide variety of possible embodiments of the present invention.

[0048] Furthermore, although process steps, method steps, algorithms, etc. may be described (in this disclosure and / or claims) in a sequential order, such processes, methods, and algorithms may be configured to operate in alternative orders. In other words, any sequence or order of steps that may be described does not necessarily require the steps to be performed in that order. Steps of processes described herein may be performed in any order that is practical. Additionally, some steps may be performed simultaneously.

[0049] Where a single device or article is described herein, it will be readily apparent that multiple devices or articles (whether or not they cooperate) may be used in place of the single device / article. Similarly, where multiple devices or articles are described herein (whether or not they cooperate), it will be readily apparent that a single device / article may be used in place of the multiple devices or articles.

[0050] The following relates generally to machine learning-based visual inspection, and more particularly to a system, method, and apparatus for visual inspection using adaptive region of interest ("ROI") segmentation. The system uses ROI segmentation to identify regions of interest ("ROIs") within an inspection image and mask or block regions of non-interest ("non-ROIs," "nROIs"). The ROIs of the inspection image are analyzed using an image analysis module that includes a machine learning model to detect defects within the inspection image (i.e., within the ROIs). By limiting the analysis to the ROIs determined by the ROI segmentation process, the system advantageously minimizes false positives and more efficiently uses computational resources.

[0051] The present disclosure also provides systems, methods, and apparatus for reducing visual inspection cycle time of an automated visual inspection process through the use of adaptive ROI segmentation.

[0052] The disclosed system advantageously limits the processed area of ​​an analyzed image to a predetermined optimal area. It reduces the likelihood of false positives. Image processing methods such as connected component analysis and region labeling allow for different tolerances or calculations to be assigned to individual regions. The system can assist with background blocking in object detection applications (thereby reducing the likelihood of false positives). The system can provide flexible definition of geometric regions (in object detection, regions must be rectangular). For areas with irregular shapes (e.g., ellipsoids), pixels outside the ROI and inside the rectangle may be ignored. For example, if an object being inspected is being held by a robotic gripper, the robotic gripper may appear in the image. Parts of the robotic gripper may be detected and interpreted as defects when analyzing the image (false positives). The system can reduce such false positives by covering the robotic gripper or parts thereof that may cause false positives.

[0053] In industrial and / or commercial environments, various parts may need to be analyzed for mechanical compatibility before delivery to or use by a customer. Similarly, each of the various parts may be affected by many different classes of defects or anomalies. Such defects may render the part defective, potentially preventing the manufacturer from selling the part while maintaining customer loyalty and / or complying with applicable laws and / or regulations. Such anomalies may not render the part so defective. However, it may be advantageous for the manufacturer to know which parts are experiencing which defects and / or anomalies. With such knowledge, the manufacturer may track the problem to a specific machine, process, supply, or precursor. With such knowledge, the manufacturer may further correct and prevent defects and anomalies revealed during the analysis.

[0054] Detailed analysis of each different part can be costly as a function of time. Human workers are generally not as adept as computers or machines at performing detailed, rote memorization tasks for extended periods of time without losing detail in the short term and without compromising job satisfaction in the long term. Therefore, it is highly advantageous for manufacturers to use automated visual inspection systems to analyze parts and detect defects and / or anomalies.

[0055] However, automated visual inspection analysis can be computationally intensive, particularly when using computer vision applications, and can require significant computer time and / or computing power. Furthermore, computing resources can be wasted by analyzing image regions that are not of interest and / or that may result in false positives. The increased processing time caused by analyzing image regions that are not of interest can significantly increase cycle time (i.e., the time to fully inspect a part) while adding little or no value to the task. Detecting false positives in visual inspection can have significant downstream negative effects, such as classifying non-defective parts as defective.

[0056] Therefore, determining ROIs and nROIs in an inspection image by segmenting the input image and masking or blocking regions of no interest may advantageously improve the efficiency of image analysis and accordingly improve the visual inspection process. In this disclosure, such segmentation of an input image that masks regions of no interest is referred to as ROI segmentation.

[0057] For example, if a mechanical part needs to be inspected for the proper assembly of part components or the presence of defects within the part, and an input image of the mechanical part is provided to an object detection model, neural network, artificial intelligence platform, or other computing device or system, the present disclosure may mask or block areas of the input image that are not relevant to the inspection of the mechanical part.

[0058] As an added benefit, adaptive ROI segmentation can restrict the area to be processed to a pre-defined optimum region, allowing different tolerances or operations to be assigned to individual areas.

[0059] As an additional advantage, adaptive ROI segmentation can enable region tracking capabilities across multiple frames or images.

[0060] As a further advantage, adaptive ROI segmentation can provide flexible definition of geometric regions on an image.

[0061] While this disclosure describes systems and methods for ROI segmentation for object defect detection and visual inspection, the systems, methods, and devices provided herein may have additional and different applications beyond those described herein, whether in the context of object defect detection and visual inspection. A computing device described herein as configured for object detection may have functionality other than object detection. In such cases, the input data may change, as may the output data, but elements of the present disclosure, such as masking regions of no interest, may function similarly.

[0062] 1, shown therein is a computer system 10 for visual inspection using adaptive ROI segmentation, according to one embodiment. The system 10 includes an adaptive ROI segmentation unit 12, which communicates with a camera unit 14, a second model unit 16, and an integrator unit 18 via a network 20.

[0063] The adaptive ROI segmentation device 12 is configured to perform adaptive ROI segmentation. The adaptive ROI segmentation device 12 may include multiple masking models. Each masking model may be trained to perform a specific masking task depending on a user's previous labeling of a specific region of a previous input image as a region of non-interest (nROI). Such user labeling may include the user masking the nROI. Masking may include detecting optimal brightness in an input image, such as the brightness of a specific object in the input image presented to the adaptive ROI segmentation device 12. Each masking model may include deep learning techniques, such as neural networks (e.g., convolutional neural networks or CNNs), and machine learning approaches. While machine learning ROI segmentation approaches require predefined features related to the desired object, neural networks do not require such definitions. In classic computer vision approaches, rule-based segmentation is used to enable the adaptive ROI segmentation device 12 to perform boundary delineation tasks using pixel contrast or color information. In deep learning models, algorithms are trained to use the overall image appearance to draw boundaries.

[0064] The adaptive ROI segmentation device 12 may also be configured to perform tasks outside the context of ROI segmentation, which may include other forms of machine learning (“ML”) or artificial intelligence tasks, or non-ML tasks.

[0065] The devices 12, 14, 16, and 18 may be server computers, node computing devices (e.g., JETSON computing devices, etc.), embedded devices, desktop computers, notebook computers, tablets, PDAs, smartphones, or other computing devices. The devices 12, 14, 16, and 18 may include a connection to a network 20, such as a wired or wireless connection to the Internet. In some examples, the network 20 may include other types of computer or communication networks. The devices 12, 14, 16, and 18 may include one or more of memory, secondary storage, a processor, an input device, a display device, and an output device. The memory may include random access memory (RAM) or a similar type of memory. The memory may also store one or more applications for execution by the processor. The applications may correspond to software modules comprising computer-executable instructions for performing processes for the functions described below. The secondary storage may include a hard disk drive, floppy disk drive, CD drive, DVD drive, Blu-ray drive, or other type of non-volatile data storage. The processor may execute applications, computer-readable instructions, or programs. The application, computer readable instructions or program may be stored in memory or secondary storage, or may be received from the Internet or other network 20 .

[0066] An input device may include any device for inputting information into devices 12, 14, 16, 18. For example, an input device may be a keyboard, keypad, cursor control device, touchscreen, camera, or microphone. A display device may include any type of device for presenting visual information. For example, a display device may be a computer monitor, a flat screen display, a projector, or a display panel. An output device may include any type of device for presenting hard copy information, such as a printer. An output device may also include other types of output devices, such as speakers. In some examples, devices 12, 14, 16, 18 may include any combination of a processor, an application, a software module, a secondary storage device, a network connection, an input device, an output device, and a display device.

[0067] While devices 12, 14, 16, and 18 are described with various components, those skilled in the art will understand that devices 12, 14, 16, and 18 may, in some examples, include fewer, additional, or different components. Furthermore, while aspects of the implementation of devices 12, 14, 16, and 18 may be described as being stored in memory, those skilled in the art will also understand that these aspects may be stored on or read from other types of computer program products or computer-readable media, such as secondary storage devices including hard disks, floppy disks, CDs, or DVDs, carrier waves from the Internet or other networks, or other forms of RAM or ROM. The computer-readable media may include instructions for controlling devices 12, 14, 16, and 18 and / or processors to perform particular methods.

[0068] Devices 12, 14, 16, 18 can be described as performing particular operations. It will be understood that any one or more of these devices may perform the operations automatically or in response to interaction by a user of the device. That is, a user of the device can manipulate one or more input devices (e.g., a touchscreen, mouse, or buttons) to cause the device to perform the described operations. In many cases, this aspect may not be described below, but will be understood.

[0069] As an example, it is described below that a device 12, 14, 16, 18 may transmit information to one or more other devices 12, 14, 16, 18. Typically, a device may receive a user interface (e.g., in the form of a web page) from the network 20. Alternatively, or in addition, the user interface may be stored locally on the device (e.g., a web page or mobile application cache).

[0070] The user interface component of the adaptive ROI segmentation device 12 may include one or more user interface elements for receiving input from a user. For example, the user interface component may provide a yes / no or similar binary option for receiving user input data indicating whether a region in the input image is a region of interest. In certain examples, the user interface may present and highlight a particular region delimited by the adaptive ROI segmentation device 12 in the input image so that the user can determine whether the region is a region of interest. In other examples, the user may define an nROI by providing input data to the ROI segmentation device 12 via the user interface.

[0071] As yet another example, when a user interface component of adaptive ROI segmentation device 12 receives particular input data (e.g., a “no” response to a question about whether a given region is a region of interest, such as by clicking a user interface element labeled “no”), adaptive ROI segmentation device 12 may be configured to incorporate the new data during training. The data so incorporated may be recorded as a training sample for a future training data set that can be used to further train adaptive ROI segmentation device 12. For example, input data provided via the user interface may be used by adaptive ROI segmentation device 12 to tag or otherwise indicate (e.g., by associating metadata) that a particular input image generated by system 10 is a training sample for a particular ROI or nROI.

[0072] The devices 12, 14, 16, 18 may be configured to receive a plurality of pieces of information from one or more of the devices 12, 14, 16, 18.

[0073] In response to receiving the information, each device 12, 14, 16, 18 may store the information in a storage database. The storage may correspond to secondary storage of one or more other devices 12, 14, 16, 18. Typically, the storage database may be any suitable storage device, such as a hard disk drive, solid state drive, memory card, or disk (e.g., CD, DVD, or Blu-ray). The storage database may also be locally connected to the device 12, 14, 16, 18. In some examples, the storage database may be located remotely from the device 12, 14, 16, 18 and accessible to the device 12, 14, 16, 18, for example, via a network. In some examples, the storage database may be comprised of one or more storage devices located at a networked cloud storage provider.

[0074] The adaptive ROI segmentation device 12 may be a dedicated machine specifically designed to perform ROI segmentation tasks, image analysis tasks, object (e.g., defect) detection tasks, object (e.g., defect) classification tasks, golden sample analysis tasks, object (e.g., defect) tracking tasks, and other related data processing tasks using inspection images captured by the camera device 14.

[0075] The camera device 14 captures image data. The image data may be of a part or object being inspected, or a portion or area thereof. The image data may include a single image or multiple images. Multiple images (frames) may be captured as video by the camera 14. The camera 14 and the object being inspected (sometimes referred to as an "inspected object" or "target object") may move relative to each other to image an area of ​​the object. For example, the object may be rotated and multiple images may be captured by the camera 14 at different positions to properly inspect from multiple angles. The camera 14 may be configured to capture multiple frames, each acquired at a different position (e.g., if the object is rotating relative to the camera 14).

[0076] Typically, the target object may be an object in which defects are undesirable. Defects in the inspected object may result in a reduction in the functional performance of the object or a larger object (e.g., a system or machine) of which the object is a component. Defects in the inspected object may reduce the visual appeal of the article. Discovering defective products may be an important step for businesses to avoid selling and using defective products and to identify the root causes associated with the defects so that such causes can be remedied.

[0077] The object being inspected may be a molded article. The object being inspected may be a manufactured item that is prone to defects occurring during the manufacturing process. The object may also be an article that derives some value from its visual appearance and where certain defects may adversely affect that visual appearance. Defects in the object being inspected may appear during the manufacturing of the object itself or during other processes (e.g., shipping, testing).

[0078] The object to be inspected may be composed of one or more materials such as metal, steel, plastic, composite, wood, glass, etc.

[0079] The object to be inspected may be uniform or non-uniform in size and shape. The object may have a curved outer surface.

[0080] The object being inspected may include multiple sections. The object sections may be further divided into object subsections. The object sections (or subsections) may be determined based on the appearance or function of the object. The object sections may be determined to facilitate easier visual inspection of the object and to better identify objects with unacceptable defects.

[0081] The sections of an object may correspond to different portions of the object having different functions. Different sections may have similar or different dimensions. In some examples, an object includes multiple different section types, with each section type appearing one or more times in the inspected object. The sections may be regularly or irregularly shaped. Different sections may have different defect specifications (i.e., tolerances for specific defects).

[0082] An object being inspected may be susceptible to multiple types or classes of defects detectable using system 10. Examples of defect types include paint, porosity, dents, scratches, sludge, etc. Defect types may vary from object to object. For example, defect types may be specific to an object based on the manufacturing process or material composition of the object. Defects in an object may be acquired during manufacturing itself or through subsequent processing of the object.

[0083] The adaptive ROI segmentation can be trained to mask different kinds or types of nROIs.

[0084] The nROI may include a non-uniform area of ​​the inspection image in which the inspected object is depicted. The non-uniform area may include components of the target article that may vary in appearance from article to article and are not of interest or relevant to visual inspection. Such relevance determination may be made in advance by a user or by the system 10 during processing.

[0085] Non-uniform areas may also include improperly illuminated areas or regions. Some visual inspection tasks require illuminating the target article. Such illumination may be translated into an inspection image of the illuminated target article. In some instances, the illumination is complex, requiring or using multiple illumination sources. The illumination may result in uneven illumination of the target article (e.g., properly illuminated or sufficiently illuminated areas and improperly illuminated or insufficiently illuminated areas). Uneven illumination can cause problems and inefficiencies in downstream image analysis processes, such as defect and anomaly detection (e.g., by resulting in false positives). By identifying and masking improperly illuminated areas that are not of interest to the visual inspection system, the system can provide improved image analysis (e.g., defect detection, anomaly detection).

[0086] Non-uniform areas may also include surfaces that could potentially produce a wide variety of anomalies and / or defects in image analysis, such as textured surfaces (e.g., cast surfaces on camshafts). Masking areas covered by such surfaces can reduce false positives and improve overall defect and anomaly detection.

[0087] The adaptive ROI segmentation device 12 includes a user interface component (or module) (e.g., a human-machine interface). During the training phase of the adaptive ROI segmentation device 12, a user may clip, mask, label, or otherwise block regions of no interest in the input image via the user interface component (or module). In other embodiments, the ROI segmentation device 12 may programmatically identify nROIs for the training samples. Rules may be hard-coded into the adaptive ROI segmentation device 12 before and / or during the training period. During the execution phase of the adaptive ROI segmentation device 12, the adaptive ROI segmentation device 12 may perform masking without further user input.

[0088] The second model device 16 receives data from the adaptive ROI segmentation device 12 via the network 20. The received data may include a mask image from the adaptive ROI segmentation device 12. For example, the mask image data may include original image data with certain regions cut out or blocked out. Such cutting, blocking, or other forms of masking may include setting all pixels within the masked region to black. All pixels within the masked region may be assigned pixel values ​​corresponding to those contained in the training data. This setting to black advantageously encourages the second model device 16 to avoid consuming computational resources by analyzing the masked region of the image.

[0089] The second model device 16 may be an object detection device 16 that includes one or more models for performing object detection tasks. The second model device 16 may include automatic image annotation software for automatically assigning metadata to mask images. For example, an inspection image may be annotated with metadata consisting of defect data generated by an object detection model, such as defect location information (e.g., bounding box coordinates, centroid coordinates), defect size data, and defect class information.

[0090] In the system 10, there may be multiple second model devices 16. Each of the second model devices 16 may receive as input an input image from the camera 14, a mask image from the adaptive ROI segmentation device 12, and / or an input image from the camera 14 that has been modified by the adaptive ROI segmentation device 12 and / or a previous second model device 16.

[0091] The integrator 18 is configured to receive the input image and the mask image modified by either the adaptive ROI segmentation device 12 or the second model device 16. The integrator 18 generates a single output, e.g., an image, in which some or all of the features or regions identified by the adaptive ROI segmentation device 12 and / or the second model device 16 are labeled, delimited, annotated, or otherwise indicated. In some examples, the ROI segmentation device 12, the second model device 16, and the integrator 18 may be implemented as a single device.

[0092] 2, shown therein is a block diagram of a computing device 1000 of system 10 of FIG. 1, according to one embodiment. Computing device 1000 may be, for example, any one of devices 12, 14, 16, 18 of FIG. 1.

[0093] Computing device 1000 includes multiple components, such as a processor 1020, which controls the operation of computing device 1000. Communication functions, including data communications, voice communications, or both, may be performed via a communications subsystem 1040. Data received by computing device 1000 may be decompressed and decoded by a decoder 1060. Communications subsystem 1040 may receive messages from and send messages to wireless network 1500.

[0094] The wireless network 1500 may be any type of wireless network, including, but not limited to, a data-centric wireless network, a voice-centric wireless network, and a dual-mode network that supports both voice and data communications.

[0095] The computing device 1000 may be a battery-powered device and may include a battery interface 1420 for receiving one or more rechargeable batteries 1440 as shown.

[0096] The processor 1020 also interacts with additional subsystems such as random access memory (RAM) 1080, flash memory 1110, display 1120 (e.g., together with a touch-sensitive overlay 1140 connected to electronic controller 1160, forming touch-sensitive display 1180), actuator assembly 1200, optional one or more force sensors 1220, auxiliary input / output (I / O) subsystem 1240, data port 1260, speaker 1280, microphone 1300, short-range communication system 1320, and other device subsystems 1340.

[0097] In some embodiments, user interaction with the graphical user interface may be performed via touch-sensitive overlay 1140. Processor 1020 may interact with touch-sensitive overlay 1140 via electronic controller 1160. Information generated by processor 1020, such as text, characters, symbols, images, icons, and other items that may be displayed or rendered on a computing device, may be displayed on touch-sensitive display 1180.

[0098] The processor 1020 may also interact with an accelerometer 1360. The accelerometer 1360 may be utilized to detect the direction of gravity or a reaction force induced by gravity.

[0099] To identify a subscriber for network access according to this embodiment, computing device 1000 may use a subscriber identity module or removable user identity module (SIM / RUIM) card 1380 inserted into SIM / RUIM interface 1400 for communication with a network (such as wireless network 1500). Alternatively, user identification information may be programmed into flash memory 1110 or implemented using other techniques.

[0100] Computing device 1000 further includes operating system 1460 and software components 1480 that are executed by processor 1020 and may be stored in persistent data storage, such as flash memory 1110. Additional applications may be loaded into computing device 1000 via wireless network 1500, auxiliary I / O subsystem 1240, data port 1260, short-range communications subsystem 1320, or any other suitable device subsystem 1340.

[0101] In use, received signals, such as text messages, email messages, web page downloads, or other data, may be processed by communications subsystem 1040 and input to processor 1020. Processor 1020 then processes the received signals for output to display 1120 or to auxiliary I / O subsystem 1240. Subscribers may further create data items, such as email messages, which may be transmitted over wireless network 1500 through communications subsystem 1040.

[0102] For voice communications, the overall operation of computing device 1000 may be similar: speaker 1280 may output audible information converted from an electrical signal, and microphone 1300 may convert the audible information into an electrical signal for processing.

[0103] 3, shown therein is a block diagram of a computer system 300 for visual inspection using adaptive ROI segmentation, according to one embodiment. System 300 can be used to visually inspect target articles. Visual inspection may include any one or more of defect detection, defect classification, and anomaly detection.

[0104] The system 300 includes a camera 304. The camera 304 captures image data of the target article 306. The image data may include a single image or multiple images. Multiple images (frames) may be captured as a video by the camera 304. To image an area of ​​the target article 306, the camera 304 and the target article 306 may move relative to one another. For example, to properly inspect the target article 306 from multiple angles, the target article 306 may be rotated, and multiple images may be captured by the camera 304 at different positions of the target article 306. The camera 304 may be configured to capture multiple frames, each captured at a different target article position (e.g., if the target article 306 is rotated relative to the camera 304). The camera 304 may be a USB 3.0 camera or an Internet Protocol ("IP") camera.

[0105] The system 300 inspects the item 306 and determines whether the item 306 is defective. The item 306 may be classified by the system 300 as defective or non-defective.

[0106] By identifying the items 306 as defective or non-defective, the inspected items may be treated differently based on the results of the visual inspection. Defective items 306 may be discarded or removed from further processing. Non-defective items 306 may continue further processing.

[0107] The camera 304 is communicatively connected to the worker node device 310 via a communication link 313 .

[0108] The camera 304 transmits the image data to the worker node device 310 via the communication link 313. In one embodiment, the camera 304 captures an image frame at the current location of the target item and transmits the image frame to the worker node device 310.

[0109] The worker node device 310 includes an adaptive ROI segmentation component 312. The adaptive ROI segmentation component 312 receives an inspection image as input and generates a mask image as output. The mask image includes an ROI and a non-ROI. nROI corresponds to a mask region of the mask image. The mask image is provided to the image analysis component 316 for analysis. The image analysis component 316 can perform defect and / or anomaly detection on the mask image. By providing the mask image to the image analysis component 316, the image analysis component 316 can function more efficiently than if it received the inspection image directly, such as by focusing image analysis on the ROI and ignoring the non-ROI.

[0110] The image analysis component 316 may include a machine learning (ML) model. The ML model is used as part of the defect detection process. The ML model may be a neural network (NN). The neural network may be a convolutional neural network (CNN). The neural network may perform an object detection (OD) task. The neural network may perform an image classification task.

[0111] In one embodiment, the image analysis component 316 is configured to generate output data that identifies the presence of defects in the image. The output data may include one or more of a defect class, a defect location, a defect size, and a defect confidence level. The defect location may be defined by a bounding box. The output data may be in the form of an annotated inspection image. For example, the annotated inspection image may include a bounding box surrounding the defect and a class label that identifies the defect type. In some embodiments, the image analysis component 316 is configured to generate output data that identifies different types of artifacts in the image, such as anomalies.

[0112] In one embodiment, the image analysis component 316 includes an object detection component for performing object detection.

[0113] The object detection component may be configured to perform an object detection process on the image data to detect defects in the image data. Typically, the object detection process determines whether a defect (or multiple defects) is present in the image data.

[0114] The object detection component may be further configured to perform a defect classification process that classifies defect types (i.e., assigns defect classes to detected defects). The defect classification can be performed on input provided from the object detection process. The defect classification process can be invoked when one or more defects are detected in the image data. The defect classification process assigns a class label (e.g., a defect name or defect type, such as "scratch") to the defects provided from the object detection process. In some embodiments, the object detection component includes an object detection model and an object classification model. The object detection model generates an object class for each detected object. The image data including the detected objects is provided to the object classification model, and the object classification model outputs the object classes. The object detection component compares the object class determined by the object detection model with the object class determined by the classification model to confirm the class label of the object. If the class label is not confirmed by the classification model, the object detection component is configured to ignore the detection.

[0115] The object detection component may use artificial intelligence, neural networks, or other means to identify objects, parts thereof, proper assemblies of those parts, and / or defects thereof within a region of interest indicated by input, such as a mask image, received from the adaptive ROI segmentation component 312. By using a mask image, the object detection component may operate more efficiently in terms of time and computational resources than if object detection were performed on an input image provided by the camera 304.

[0116] In one embodiment, the image analysis component 316 includes a golden sample component for performing golden sample (GS) image analysis. The golden sample component may or may not be a generated golden sample component (for generating GS images from inspection images). A golden sample component that is not a generated GS component may have a bank of GS images from which to search for an appropriate GS image.

[0117] The golden sample component may also include an image comparison component (not shown). The image comparison component analyzes the GS image and the inspection image to identify artifacts corresponding to differences between the GS image and the inspection image. Typically, the GS image represents a "perfect" or "clean" image, so artifacts are present in the inspection image but not in the GS image. In one embodiment, the inspection image and the GS image are each provided to a pre-trained CNN to generate respective feature maps. The feature maps are compared / analyzed to identify artifacts. In another embodiment, the GS image and the inspection image are compared using matrix subtraction or pixel-to-pixel grayscale subtraction to generate an output that identifies artifacts. The results of the image comparison component can be used in subsequent processing operations, such as detecting anomalies or identifying new types or classes of defects. For example, the output of the golden sample component (identified differences or objects) can be compared with the output of the object detection component (detected objects).

[0118] The golden sample component may include an image classification model. The image classification model receives an input image containing the artifact detected by the image comparison component. The input image may be a cropped version of the inspection image containing the artifact. The image classification model is configured to determine a class label and assign the class label to the artifact. The image classification model may be a binary classifier configured to assign a defect label or an anomaly label to the input image.

[0119] The golden sample component may use artificial intelligence, neural networks, or other means to create an idealized representation of the region of interest indicated by an input, such as a mask image received from the adaptive ROI segmentation component 312. This idealized representation is referred to as a “golden sample” or “golden sample image.” This golden sample may represent an image of an object or part that is free of defects or improper assembly, so that a comparison of the golden sample of the mask image with the mask image itself can be made. By using a mask image, the golden sample component may operate more efficiently in terms of time and computational resources than if it created the golden sample for an input image provided by the camera 304.

[0120] The object detection component and the golden sample component may be communicatively coupled via a communication link within the ROI segmentation component 312. The communication link may include an application programming interface (API) or the like.

[0121] The object detection component and the golden sample component may each have their own designated hardware components (not shown). For example, the object detection component may run on a first embedded device and the golden sample component may run on a second embedded device. The embedded devices may be embedded devices specially configured to perform artificial intelligence type tasks. In one embodiment, the embedded device is a JETSON box.

[0122] In one embodiment, the image analysis component 316 includes an OD component, a GS component, and an integration component. The integration component may be implemented in the integration device 18 of FIG. 1. The integration component is configured to receive output data from the OD component (e.g., describing defects found in the inspection image) and the GS component (e.g., describing defects or anomalies found in the inspection image). The integration component compares the OD output data with the GS output data. The comparison may include comparing location data of the defects / anomalies in each output. The integration component may confirm defects found whose location data matches (e.g., within a threshold) between the OD output and the GS output.

[0123] If provided with a mask image from the adaptive ROI segmentation component 312, the image analysis component 316 may instead perform the object detection function in the object detection component and / or the golden sample function in the golden sample component, after which the integration component may perform the comparison function.

[0124] Either or both of the object detection component and the golden sample component may be housed within one or more image analysis components 316. The image analysis component 316 may include additional models and / or components for analyzing the input image and / or the mask image.

[0125] The integration component may integrate information and decisions made regarding the input image and / or mask image to generate an output image. The output image is received by the client device 338 and displayed to the user. The output image may be an annotated inspection image including visually identified defects or anomalies detected by the image analysis component 316. The output image may be displayed to the user on the display 346 of the client device 338. Through the client device 338, the integration component may provide all or only a portion of the information generated by the object detection component, the golden sample component, and / or any additional models and / or components of the image analysis component 316. The output to the user at the client device 338 may include either the input image or the mask image annotated thereon or provided with such information. In further embodiments, the output to the user at the client device 338 may include information presented independently of the input image or the mask image.

[0126] The devices, components, and databases of system 300 communicate with each other through communication links, such as communication link 313, shown here connecting camera 304 and adaptive ROI segmentation component 312. Those skilled in the art will understand that such communication links may exist between more, fewer, and all of the devices, components, and databases of system 300.

[0127] The system 300 further includes a PLC device 320. The PLC device 320 is communicatively connected to the worker node device 310 via a communication link 322.

[0128] The PLC device 320 is configured to control the manipulation and physical processing of the target item 306. This can be done by sending and receiving control instructions to an item manipulation unit (not shown) via a communication link 321. Such manipulation and physical processing may include rotating or moving the target item 306 for imaging, and moving the target item 306 into and out of an inspection area. An example of an instruction that the PLC 320 sends to the item manipulation unit via the communication link 321 may be "Rotate the target item 'n' degrees." In some examples, the sending of such an instruction may depend on information received from the worker node device 310 (e.g., the object detection component 314).

[0129] The PLC 320 may store defect tolerance data. The defect tolerance data may include a defect class identifier specific to a particular defect class and one or more tolerance values ​​linked to the defect class identifier. In other embodiments, the defect tolerance data may be stored in another device, such as the operator node device 310. The defect tolerance data may be stored in a defect tolerance database. The defect tolerance data in the defect tolerance database may be referenced using the defect class identifier to facilitate retrieval of tolerance data values ​​for comparison with data generated by the operator node device 310 (e.g., via components 314, 316).

[0130] For example, in one embodiment, PLC 320 is configured to receive data indicative of the results of the defect detection process from operator node device 310 via communication link 322. For example, if a defect is detected by object detection component 314, the defect data may be transmitted to PLC 320. PLC 320 stores defect tolerance data. PLC 320 analyzes the defect data in light of the tolerance data and determines whether target item 306 is defective (e.g., “NG”) or within tolerance (e.g., “OK”). PLC 320 may transmit a signal indicative of the result of the tolerance analysis to operator node device 310. If PLC 320 determines that the defect data is out of tolerance, PLC 320 may stop inspection of target item 306 and initiate a process for removing the defective target item and preparing a new target item. PLC 320 may generate a control signal to stop inspection of the target item and transmit the control signal to an actuator responsible for manipulating target item 306.

[0131] If the object detection component 314 does not detect any defects in the inspection image, the worker node device 310 (e.g., via the object detection component 314) sends a signal (e.g., "OK") to the PLC 320 indicating the result of the object detection process indicating that no defects were found in the image. Upon receiving the OK message, the PLC 320 sends a control signal to an actuator or manipulator of the target item 306 to adjust the current inspection position of the target item 306 (e.g., rotate the target item 306 "n" degrees).

[0132] In other embodiments, the defect tolerance data may be stored in the operator node device 310, and the tolerance analysis may be performed by the operator node device 310. The operator node device 310 may then send a signal to the PLC 320 indicating whether the target item is defective.

[0133] The system 300 further includes an operator device 324. The operator device 324 is communicatively connected to the worker node device 310 via a communication link 326.

[0134] The operator device 324 includes a user interface component (or module) (e.g., a human-machine interface). The operator device 324 receives data from the worker node device 310 via a communication link 326. The received data may include output data from the image analysis component 316 of the worker node device 310. For example, the output data may include an annotated inspection image including artifact data. The artifact data may include location information (e.g., coordinates, bounding box) and label information so that artifacts (e.g., defects, anomalies) in the inspection image identified by the worker node device 310 can be visually identified in a display image.

[0135] The worker node device 310 or operator device 324 may include automatic image annotation software for automatically assigning metadata comprising data generated by the image analysis component 316 to the digital inspection images.

[0136] The operator device 324 provides output data from the worker node device 310 to a user interface component that generates a user interface screen that displays the annotated inspection image. For example, the inspection image may be annotated with metadata including defect data generated by the component, such as defect location information (e.g., bounding box coordinates, centroid coordinates), defect size data, defect class information, etc.

[0137] The user interface component of the operator device 324 may render one or more user interface elements for receiving input from the operator. For example, the user interface component may provide a yes / no or similar binary option for receiving user input data indicating a selection of the option. In a particular example, the user interface may present and highlight a particular object detected by the worker node device 310 within the annotated inspection image and ask (and receive corresponding input from) whether the object is an anomaly.

[0138] Depending on the input data received from the user, the annotated inspection image (or a portion thereof) may be routed differently within the system 300. For example, when a user interface component of the operator device 324 receives particular input data (e.g., a "no" response to a question about whether a given artifact is anomalous, such as by the user clicking a user interface element labeled "no"), the operator device 324 or the worker node device 310 may be configured to transmit the annotated inspection image (or a subset of the image data) to an ML model training database, such as training database 330, via communication link 332 or 333, respectively. The data received by the ML model training database 330 may be recorded as training samples for a future training dataset that can be used to further train one or more artificial intelligence components of the worker node device 310.

[0139] The operator device 324 may be used in the training phase of the ROI segmentation component 312. For example, a test image may be displayed on a user interface (training sample annotation user interface) of the operator device 324. A user may input data identifying an nROI within the test image via the user interface. In some examples, a user may identify an nROI by defining an ROI within the image (i.e., an nROI is a region of the image not identified as an ROI). The resulting images with nROIs identified constitute training samples that can be added to the training database 330 and used to train (or retrain or update) the ROI segmentation component 312. Once trained, the system 300, and in particular the adaptive ROI segmentation component 312, can perform adaptive ROI segmentation without further user input by relying on the training database 330.

[0140] The system 300 further includes a server node device 334. The server node device 334 is communicatively coupled to the worker node device 310 via a communication link 336. In particular, the server node device may communicate with the image analysis component 316 of the worker node device 310 via the communication link 336. The server node device 334 may include a Jetson device or the like.

[0141] The server node device 334 receives visual inspection data from the worker node device 310. The visual inspection data includes output data (or "defect data") from the image analysis component 316. The defect data may include whether or not defects were found, a unique defect identifier for each detected defect, the number of defects found, whether or not the target item is defective, the location of the defects (defect location data such as bounding box coordinates), a defect class identifier, etc. The server node device 334 includes a visual inspection analysis component configured to analyze the received defect data.

[0142] The server node device 334 is communicatively connected to the client device 338 via a communication link 340. In some examples, the client device 338 may include the server node device 334 (i.e., the server node device 334 is a component of the client device 338).

[0143] The server node device 334 is communicatively connected to an analysis database 342 via a communication link 344. The analysis database 342 stores the analysis data as well as the visual inspection output data (e.g., defect data) from the worker node device 310.

[0144] The defect data may be stored such that a database record is created and maintained for each inspected item 306. The record includes an item identifier, which may be captured from the item itself (e.g., a code on the item captured by a camera) or may be automatically generated by the server node device 334. Various defect data may be associated with or linked to the database record for the inspected item 306. Each defect may be assigned a unique identifier to which other data regarding the defect may be linked.

[0145] Analytical data may be generated from the visual inspection data by the server node device 334. The analytical data may be generated by applying statistical analysis techniques to the visual inspection data. The analytical data may provide an operator or other user with insight into the determinations made by the system 300 across multiple target articles 306.

[0146] The client device 338 includes a user interface component configured to provide a graphical user interface via a display 346. The user interface component receives the analysis data from the server node device 334 and displays the analysis data via the graphical user interface on the display 346. In some examples, the server node device 334 and the user interface component are configured to update the graphical user interface in real time as the server node device 334 receives the visual inspection data from the worker node device 310.

[0147] Referring now to FIG. 4, shown therein is a computer system 400 for visual inspection using adaptive ROI segmentation, according to one embodiment. The computer system 400 may be implemented by one or more devices of the computer system 10 of FIG. 1. For example, the components of the computer system 400 may be implemented by any one or more of the adaptive ROI segmentation device 12, the second model device 16, and the integrating device 18 of FIG. 1. The computer system may be implemented by one or more devices of the system 300 of FIG. 3. For example, the components of the computer system 400 may be implemented by the worker node device 310 of FIG. 3.

[0148] The system 400 includes a processor 402 for executing software models and modules.

[0149] The system 400 further includes a memory 404 in communication with the processor 402 for storing data, including output data from the processor 402 .

[0150] The memory 404 stores inspection image data 410 corresponding to the inspection image, which may be generated and provided by the camera 304 of FIG.

[0151] The processor 402 includes a training module 406 for training an adaptive ROI segmentation model (e.g., ROI segmentation model 413). The training module 406 receives user input indicating nROIs (or ROIs) in an input image to use in training the ROI segmentation model. The user input is stored in memory 404 as training sample annotation data 408. The training module 406 generates training images by annotating the input image with the training sample annotation data 408. The training images are stored in memory 404 as training image data 407 (e.g., as part of a training dataset comprising a plurality of training images). In some examples, the training module 406 can perform the annotation of the training images programmatically or automatically without user input.

[0152] The processor 402 further includes an adaptive ROI segmentation module 412. A training phase exists in which an adaptive ROI segmentation model 413 is trained using a training module 406 in cooperation with the adaptive ROI segmentation module 412. The memory 404 stores training image data 407. The training module 406 is configured to display the training image data 407 on a user interface (not shown). A user reviews the training image data 407 and inputs training sample annotation data 408 to the training module 406 via the user interface. The training sample annotation data 408 indicates masked non-ROI regions in the training image data 407. The training module 406 annotates the training image data 407 using the training sample annotation data 408 to generate annotated training image data 409 (masked training image data). The annotated training image data 409 is stored in the memory 404. A user can manually provide training sample annotation data 408, for example, by blacking out non-ROI regions with known photo alteration techniques and / or software.

[0153] In one embodiment, the training module 406 may be configured to train the adaptive ROI segmentation model 413 so that the adaptive ROI segmentation model 413 performs ring segmentation. The training module 406 may include software code configured to find two circles having a predetermined diameter and extract the inner and outer circles using a circle detection algorithm. The training module 406 uses the extracted inner and outer circles to draw an ROI mask on the training image data 407. Thus, the trained adaptive ROI segmentation model 413 (e.g., the trained network) can generalize to draw such rings without further reliance on a circle detection algorithm (which, for example, may be slower than a deep learning network and may fail on complex tasks where multiple circular objects are present).

[0154] The training module 406 is further configured to perform a training process using the training image data 407 and the annotated training image data 409. The training image data 407 and the corresponding annotated training image data 409 may be used by the training module 406 to learn a model configuration for the autoencoder model and, through the learning process, generate a trained ROI segmentation model 413. The model training process employed by the training module 406 may be the same as or similar to the model training process of a typical autoencoder.

[0155] The adaptive ROI segmentation module 412 generates a mask image. The nROI in the masked image is cut out from the image by providing the output image to other image processing algorithms. The adaptive ROI segmentation module 412 may generate a masked output image that includes the blocked nROI by setting pixels in the nROI to black (black pixels are nothing and contain no content).

[0156] The adaptive ROI segmentation module 412 includes a trained ROI segmentation model 413. The trained model 413 may be generated by the training module 406. The adaptive ROI segmentation module 412 may function similarly to the adaptive ROI segmentation device 12. The trained ROI segmentation model 413 may be structurally and functionally similar to an existing autoencoder. The trained ROI segmentation model 413 may be configured to downsample the test image data 410 using a convolutional layer and map the test image data 410 to a latent vector. The trained ROI segmentation model 413 may then upsample the test image data 410 and generate an output image.

[0157] Through communication between memory 404 and processor 402, the test image data 410 is provided to an adaptive ROI segmentation module 412 and provided as input to an ROI segmentation model 413. The ROI segmentation model 413 generates as output segmentation output data 414. The segmentation output data 414 is stored in memory 404. The segmentation output data 414 may comprise a masked image in which the n ROIs have been masked. For example, the segmentation output data 414 may be an image in which the n ROIs have been rendered in blocks (similar to the annotated training image data 409).

[0158] The segmentation output data 414 includes ROI data 416 that indicates one or more regions of interest (ROI) within the inspection image data 410. The segmentation output data 414 further includes nROI data 418 that indicates one or more regions of no interest (and therefore masked) in the inspection image data 410. Either or both of the ROI 416 and nROI 418 may be in the form of an input image in which the ROI has been highlighted and / or in which the nROI has been cropped, blocked, blacked out, or otherwise masked.

[0159] The processor 402 further includes an image analysis module 420. The image analysis module 420 includes one or more machine learning models 421, such as a neural network, for detecting objects in an input image. The objects that the machine learning models 421 are trained to detect may be defects. For example, the machine learning models 421 may be configured to detect or classify one or more categories or types of defects in the input image. The machine learning models 421 may be an object detection model. The machine learning models 421 may be an image classification model. The machine learning models 421 may be a neural network, such as a CNN, configured to detect features in an input image (e.g., the mask inspection image 414).

[0160] Image analysis module 420 receives and analyzes mask inspection image 414 to detect objects / artifacts in the mask inspection image using machine learning model 421. In particular, analysis of mask inspection image 414 is limited to ROI 416 determined by ROI segmentation model 413.

[0161] Through communication between the memory 404 and the processor 402 , the inspection image data 410 , the ROI data 416 , and / or the nROI data 418 may be provided to an image analysis module 420 .

[0162] Image analysis module 420 uses machine learning model 421 to generate image analysis output data 422. For example, in embodiments in which machine learning model 421 includes an object detection model, image analysis output data 422 includes object detection output data.

[0163] In some examples, the image analysis output data 422 may be an annotated inspection image (eg, annotated with detected object data, as described below).

[0164] The image analysis output data 422 includes detected object data 424. The detected object data 424 describes one or more objects identified within the ROI 416 (mask inspection image 414) by the image analysis module 420. The objects may be defects. The objects may be anomalies.

[0165] The detected object data 424 includes object identifier data 426. The object identifier data 426 may include a unique identifier (generated by the image analysis module 420) for each detected object.

[0166] The detected object data 424 may include object class data 428. The object class data 428 may include an assignment of a class label to each detected object. The class label may correspond to a particular category, class, or type of defect.

[0167] Detected object data 424 may include object position data 430. Object position data 430 may include position data for each detected object that defines the location of the detected object within the inspection image. For example, object position data 430 may be in the form of a bounding box that encloses the detected object within ROI 416, mask inspection image 414, or inspection image 410.

[0168] The detected object data 424 may include object confidence data 432. The object confidence data 432 indicates the confidence of each detected object.

[0169] The detected object data 424 may include object size data 434. The object size data may include size data that identifies the size of each detected object.

[0170] 5A, shown therein is one embodiment of a defect detection pipeline 500 using ROI segmentation, which is implemented by a visual inspection system, such as system 300 of FIG. 3 or computer system 400 of FIG. 4.

[0171] At 502, camera 304 images target item 306. The imaging operation produces inspection image 504 (e.g., inspection image 410 in Figure 4).

[0172] The test image 504 is provided to an adaptive ROI segmenter module 506 for performing adaptive ROI segmentation and masking of the nROI (e.g., nROI 418 of FIG. 4). The ROI segmenter module 506 may be the ROI segmentation module 412 of FIG. 4.

[0173] Adaptive ROI segmenter module 506 identifies and masks nROIs in the test image to generate mask test image 508. Mask test image 508 is test image 504 with non-ROI regions cut out or otherwise blacked out. Mask test image 508 allows subsequent analysis to be more efficiently directed to the ROI.

[0174] The mask inspection image 508 is provided to a first image analysis module 510 for performing defect detection. The first image analysis module 510 is configured to analyze only the ROIs, as identified by the ROI segmenter module 506, and ignore the nROIs.

[0175] First image analysis module 510 generates output image 512 including defect data. The defect data identifies defects detected in the mask inspection image. The defect data may be detected object data 424 in FIG. 4. Output image 512 may take the form of an annotated version of either inspection image 504 or mask inspection image 508 with annotations (such as labels, class labels, bounding boxes, and / or coordinates) that describe the detected defects.

[0176] The pipeline 500 includes an optional component 514 .

[0177] In optional component 514, mask inspection image 508 is also provided to a second image analysis module 516 for performing defect detection. Second image analysis module 516 is configured to analyze only the ROIs, as identified by ROI segmenter module 506, and ignore non-ROIs.

[0178] Second image analysis module 516 generates output image 518 including defect data. The defect data identifies defects detected in the mask inspection image. The defect data may be detected object data 424 in FIG. 4. Output image 518 may take the form of an annotated version of either inspection image 504 or mask inspection image 508 with annotations (such as labels, class labels, bounding boxes, and / or coordinates) that describe the detected defects.

[0179] The output images 512, 518 are provided to a comparison module 520 for comparing the defects detected by the first image analysis module 510 and the second image analysis module 516. The comparison module 520 may be configured to confirm the defect in the inspection image 504 and / or the mask inspection image 508 if the defect is present in both the output images 512 and 518, as determined by the comparison module 520. The comparison may be performed using defect location data (e.g., object location data 430 of FIG. 4 ) of the defect in the respective output images 512, 518. If the comparison module 520 confirms the defect, it may further classify the defect.

[0180] The first image analysis module 510 and the second image analysis module 516 may perform defect detection through different techniques and / or models. For example, the first image analysis module may perform defect detection according to an object detection technique and / or model, and the second image analysis module may perform defect detection according to a golden sample technique and / or model.

[0181] 5B, shown therein is a training pipeline 501 of a visual inspection system for defect detection using ROI segmentation, according to one embodiment. The training pipeline 501 may be implemented by the training module 406 of FIG. 4.

[0182] In the pipeline 501, the inspection image 504 is provided to a preprocessing module 522, which generates a preprocessed image 524. The preprocessed image 524 may take the form of an annotated version of the inspection image 504, with annotations identifying the ROIs and nROIs. The annotations may be provided by a user in a supervised or semi-supervised manner. Alternatively, the annotations may be provided programmatically and may be based on the position or rotation of the object (being inspected) in the inspection image 504. For example, an object may rotate or move during inspection, and multiple images may be acquired. In such cases, metadata indicating the object's position or rotation may be associated with the inspection image, so that the position or rotation at which the image was taken is known. This position information may be leveraged by the preprocessing module to automatically identify the ROIs and nROIs.

[0183] The preprocessing module 522 may perform preprocessing by providing the inspection image 504 to a user (e.g., by displaying the inspection image 504 in a user interface). The user can then indicate the ROIs and nROIs within the inspection image 504 by providing input data to the user interface (e.g., by manually cropping or blacking out the nROIs). Such user cropping or blacking out can be performed using known photo-manipulation techniques and / or software (e.g., Photoshop®, etc.). In other cases, the preprocessing module 522 may be configured to automatically preprocess the image, for example, according to ML and / or CNN techniques. The preprocessed image 524 has the nROI cropped or otherwise blacked out.

[0184] At 526, the preprocessed image 524 is added to a training dataset 528 for training an ROI segmentation model.

[0185] The training data set 528 is used to construct a training module 530 for training the ROI segmenter model and generating the trained ROI segmenter model 507 .

[0186] The trained ROI segmenter model 507 can be incorporated into the adaptive ROI segmenter module 506 .

[0187] Pipelines 500 and 501 may intelligently crop test image 504 to remove nROIs and may limit the input to trained model 507 of ROI segmenter module 506.

[0188] Non-ROI masking is advantageous in downstream algorithms that require perfect illumination of the object's surface in the inspection image to perform accurate inspection.

[0189] As a further example, when adaptive ROI segmentation is used to generate stitched images from an area scan camera, masking can be advantageous: non-overlapping areas of the images may be cropped out for addition to the output image.

[0190] As a further example, in medical applications, it may be necessary to run a diagnostic algorithm (e.g., image analysis using a machine learning model) on a predetermined region of tissue. For example, a user, such as a medical professional, may acquire an image of an individual's tissue for the purpose of diagnosing the individual's condition (i.e., determining whether the individual has a particular medical condition). The image may be masked using the adaptive ROI segmentation techniques of the present disclosure. In particular, the ROI determined by ROI segmentation may correspond to one or more predetermined regions or regions of the tissue that are of interest for diagnosis, and the nROI may correspond to other regions or regions of the tissue that are not of interest (and are therefore masked). The masked image may be provided to a downstream image analysis process that uses a machine learning model (e.g., object detection, classification). Analysis by the machine learning model is limited to the ROI. Doing so may reduce the chance of a false positive, such as detecting something in the image within the nROI.

[0191] The ROI of the object being inspected in inspection image 504 may vary depending on the application. For example, in a visual inspection of a camshaft, the ROI may be a uniformly illuminated region of the camshaft image (e.g., the center of the camshaft). Anomaly detection tasks that may be performed downstream of adaptive ROI segmentation may have different requirements for different regions of inspection image 504. Furthermore, not all pixels of inspection image 504 may be used for inspection, and masking pixels not used for inspection may reduce false positives.

[0192] 6, shown therein is a diagram of example images 602a, 602b, 604a, 604b and logic 606 corresponding to the systems and apparatus of the present disclosure, according to one embodiment. Images 602a, 602b, 604a, 604b may be processed by computer system 400 of FIG. 4 using logic 606. Logic 606 represents a trained ROI segmentation model, such as model 413 of FIG. 4.

[0193] Sample input image 602a illustrates an input image of a camshaft under inspection (e.g., inspection image data 410) before adaptive ROI segmentation is applied. Specifically, input image 602a depicts a journal portion of the camshaft.

[0194] An input image 602a is provided to an ROI segmentation model 606, which generates an output image 602b.

[0195] Output image 602b is a masked inspection image. Output image 602a includes an ROI and nROI. nROI has been masked with black pixels by model 606. The masked region in output image 602b is illustrated using a hatch pattern (representing black pixels). Masking the nROI is advantageous for limiting subsequent processing of the inspection image by other models to only the ROI.

[0196] Output image 602b is a journal with non-uniform regions masked. The seal rings in output image 602b are masked. Adaptive ROI segmentation is configured to mask the seal rings because their appearance may vary from camshaft to camshaft. If the seal rings were not masked in the output image, they could adversely affect downstream image analysis processes such as anomaly detection (e.g., by generating false positives). Improperly lit regions in output image 602b are masked. The cast surface is also masked in output image 602b. The cast surface has texture, and this texture, when presented to downstream image analysis processes such as anomaly detection, can cause false positives, i.e., unwanted detections.

[0197] Sample input image 604a is another example input image of a camshaft under inspection before ROI segmentation is applied. Specifically, input image 604a depicts the thrust portion of the camshaft. Input image 604a is provided to ROI segmentation model 606, which generates output image 604b, which (like output image 602b) is a mask image.

[0198] The output image 604b includes the ROI and nROI. The nROI is masked with black pixels by the model 606. The masked area in the output image 604b is illustrated using a hatch pattern (representing black pixels). The trained ROI segmentation model 413 is trained to generate an enhanced image of the thrust area (due to poor lighting conditions in the input image 604a). In the output image 604b, the cast surface present in the input image 604a is masked. As mentioned above, when the cast surface is presented for downstream image analysis, such as anomaly detection, unwanted detections may occur due to the texture of the cast surface.

[0199] The model 606 is a denoising autoencoder configured to learn a segmentation task. The layers, cost function, and weights of the denoising autoencoder 606 can be different for different inspection tasks. For example, when an input image 602a is used to generate an output image 602b, the content of the input image 602a is downsampled to a latent space (code). During the reconstruction phase, non-ROIs in the image data are converted to black pixels.

[0200] Referring now to FIG. 7, shown therein is a diagram of an exemplary input and output image pair processed by the ROI segmentation module of the present disclosure, according to one embodiment. The input and output image pair depicts a camshaft being rotated during inspection, with images captured at different points in the rotation. The input and output images may be generated and processed by the system 300 of FIG. 3. The input image may be the inspection image data 410 of FIG. 4. The output image may be the segmentation output data 414 of FIG. 4.

[0201] Each of columns 702a, 704a, and 706a includes an input image (e.g., test image data 410) before adaptive ROI segmentation is applied, and after adaptive ROI segmentation is performed, for example, according to logic 606 of FIG. 6 or adaptive ROI segmentation module 412 of FIG. 4. Columns 702b, 704b, and 706b include output images produced by the ROI segmentation process applied to the corresponding input images (e.g., the input image in the first row of 702a corresponds to the output image in the first row of 702b). The output images are masked test images (i.e., masked versions of the corresponding input images). The masked test images include masked regions corresponding to nROIs. The ROIs are unmasked. The nROIs in output images 702b, 704b, and 706b are masked with black pixels. 7, the masked regions in the output images 702b, 704b, and 706b are illustrated using a hatch pattern (representing black pixels). This masking can advantageously limit subsequent processing by other models to only the ROI within each output image 702b, 704b, and 706b.

[0202] As noted above, the input / output image pairs depict the camshaft being inspected as it rotates. Therefore, the image pairs show how the ROI may change or move as the camshaft rotates. In columns 702a and 702b, the bright areas of the camshaft are tracked across multiple frames. In columns 704a and 704b, the vacuum slot openings are masked across multiple frames without missing any areas of the work surface. Additionally, the sensor rings are also masked because they are not areas requiring inspection. In columns 706a and 706b, the VTC oil holes and stepper motor cutouts are masked across multiple frames.

[0203] The system 10 for visual inspection may be used to visually inspect machined surfaces. Machined surfaces may be particularly suitable for visual inspection using machine learning or computer vision processes, as described herein. The system 10 may also be used to visually inspect articles that include machined surfaces. The article may include other parts, components, regions, etc., other than the machined surface being inspected. Such other parts, components, regions, etc., may be captured and present in image data of the article used for visual inspection. Additionally, other aspects that are not part of the article may also be included in the image data. The visual inspection may be focused on the machined surfaces.

[0204] 8, shown therein is a visual inspection processing pipeline 800 for golden sample image analysis with adaptive ROI segmentation, according to one embodiment. Pipeline 800 can be implemented using, for example, system 300 of FIG. 3 or computer system 400 of FIG. 4.

[0205] The pipeline 800 begins with an input image 802 (e.g., inspection image data 410), which may be an inspection image of the target article 306 captured by the camera 304.

[0206] An input image 802 is provided to an adaptive ROI segmenter (eg, adaptive ROI segmentation module 412 of FIG. 4). The adaptive ROI segmenter 804 generates masked input image data 806.

[0207] The masked input image data 806 includes a mask that covers nROIs in the input image data 802 identified by the ROI segmenter 804. Masking of the nROIs can be done by setting the pixels of the nROIs in the input image 802 to black. This may include, among other things, setting the pixels within the nROIs to black or setting all pixels in the image that are outside the ROI to black.

[0208] The masked input image data 806 is provided to a Generative Model 810. The Generative Model 810 generates a masked golden sample image 812. Advantageously, because the nROIs are masked in the masked input image data 806, the Generative Model 810 does not perform generation on those regions. Thus, the Generative Model 810 can operate more efficiently and effectively by avoiding unnecessarily processing regions of no interest as identified by the Adaptive ROI Segmenter 804.

[0209] The masked input image 806 and the masked golden sample image 812 are each provided to an image comparison module 808 .

[0210] The image comparison module 808 performs a direct image comparison on the masked input image data 806 and the masked golden sample image 812 (i.e., the masked input image data before and after passing through the generative model 810) and generates comparison output data 814. The comparison output data 814 may include one or more detected objects (or artifacts). A detected object in this context refers to an object or artifact that is present in the masked input image 806 but not in the masked golden sample image 812. In other words, the detected object represents a difference detected between the images 806, 812.

[0211] The image comparison module 808 may compare the images 806, 812 on a pixel-by-pixel basis. In one embodiment, direct image comparison is performed using matrix subtraction or pixel-to-pixel grayscale subtraction. Advantageously, such comparisons may proceed more efficiently because the presence of masking causes the nROI to be ignored, eliminating the need for the image comparison module 808 to perform comparisons with respect to such regions.

[0212] The comparison output data 814 may be provided to a classification model 816 for classification (e.g., defect, part identification) of objects found by the image comparison module 808. In some examples, the objects identified by the image comparison module 808 are segmented from the masked input image, and the segmented image containing the objects is provided to the classification model 816 for image classification of the segmented image.

[0213] The classification model 816 generates classification output data 818. The classification output data 818 includes class labels assigned to the classified objects (e.g., defect type, defect vs. anomaly). The classification output data 818 may be advantageously generated more efficiently and effectively through avoiding unnecessary processing and computation due to the presence of masking provided by the adaptive ROI segmenter 804. Furthermore, the adaptive ROI segmenter 804 may advantageously reduce the load required of the classification model 816 by preventing false positive detections within nROIs from being sent to and processed by the classification model 816 (because such nROIs are masked).

[0214] Referring now to FIG. 9, shown therein is a method 900 of visual inspection using the visual inspection processing pipeline 800 of FIG. 8, according to one embodiment.

[0215] At 902, an input image 802 is acquired.

[0216] At 904 , the input image 802 is provided to an adaptive ROI segmenter 804 .

[0217] At 906, an autoencoder output is generated and a distinct mask is derived from the autoencoder output.

[0218] At 908, a separate mask is applied to the input image 802 to block out regions of no interest in the input image 802. The result is masked input image data 806.

[0219] At 910 , the masked input image data 806 is input into a generative model 810 to generate a masked golden sample image 812 .

[0220] At 912, the image comparison module 808 performs an image comparison between the masked input image data 806 and the masked golden sample image 812. Such a comparison can be performed on a pixel-by-pixel basis in the region of interest identified by the adaptive ROI segmenter 804, limited to excluded pixels (i.e., pixels outside the mask).

[0221] At 914 , comparison output data 814 is generated that includes the detected artifacts or differences between the masked input image data 806 and the masked golden sample image 812 .

[0222] At 916, each of the detected artifacts or discrepancies is provided to a classification model 816, which is configured to assign them a class label. The classification model 816 may be a binary classification model.

[0223] At 918, classification output data 818 is generated. The classification output data 818 may include assigned class labels.

[0224] Because regions of interest are identified and consequently masked by adaptive ROI segmenter 804, processing may be advantageously limited to only those portions of input image 802 (and correspondingly masked portions of input image data 806 and masked golden sample image 812) deemed relevant by an operator and / or the logic of adaptive ROI segmenter 804. Thus, method 900 may promote greater efficiency and effectiveness in computer processing.

[0225] Referring now to Figure 10, therein is shown a visual inspection pipeline 801 for golden sample image analysis with adaptive ROI segmentation, according to one embodiment. Pipeline 801 is an alternative configuration of pipeline 800 shown in Figure 8. Therefore, similar reference numbers are used. Only the differences will be described.

[0226] In pipeline 801, input image 802 is provided to generative model 810 to generate golden sample image 820. Thus, golden sample image 820 is generated and then provided to adaptive ROI segmenter 804 to obtain masked golden sample image 812. This process contrasts with the process disclosed for system 800, in which masked golden sample image 812 is generated from masked image data 806.

[0227] Referring now to Figure 11, shown therein is a method 1100 by which system 801 operates, according to one embodiment. In Figure 11, steps similar to steps in Figure 9 are indicated by the same last two reference numbers (e.g., 1102, 902). For example, steps 1102-1108 function similarly to steps 902-908 in Figure 9.

[0228] In method 1100, 910 is replaced by 1120. Similar to system 801 of Figure 10, masked golden sample image 812 is generated by applying the individual masks of adaptive ROI segmenter 804 to golden sample image 820. Optionally, golden sample image 820 is generated from input image data 802 by generative model 810.

[0229] Referring now to FIG. 12, shown therein is a visual inspection processing pipeline 1200 for golden sample image analysis with adaptive ROI segmentation according to one embodiment.

[0230] The pipeline 1200 begins with an input image 802, similar to FIG.

[0231] An input image 802 is sent to an adaptive ROI segmenter 804. The adaptive ROI segmenter 804 generates masked input image data 806.

[0232] The masked input image data 806 is provided as input to a first pre-trained convolutional neural network ("CNN") 822. The first CNN 822 processes the masked input image data 806 and generates a first feature map. The first feature map is provided to a feature map analysis module 826.

[0233] The masked input image 806 is also provided to a generative model 810, which generates a masked golden sample image 812. In other embodiments, the input image 802 may be provided directly to the generative model 810 to generate the golden sample image, which is provided to the adaptive ROI segmenter 804 to generate the masked golden sample image 812.

[0234] The masked golden sample image 812 is provided as input to a second pre-trained CNN 824. The second CNN 824 may have the same configuration as the first CNN 822. The second CNN 824 and the first CNN 822 may be separate instances of the same pre-trained CNN. In some examples, the second CNN 824 is not used, and the masked golden sample image 812 is provided to the first CNN 822.

[0235] The second CNN 824 processes the masked golden sample image 812 and generates a second feature map, which is provided to a feature map analysis module 826.

[0236] The feature map analysis module 826 performs a feature map comparison of the first and second feature maps and identifies differences (feature map differences). In one embodiment, the feature map comparison uses a downsampled version of the image, where each pixel contains a vector equivalent to a similarly sized vector for the same pixel from the feature map of the input image.

[0237] The feature map analysis generates an output that includes the differences (features) identified between the first and second feature maps. Typically, the identified features correspond to features or objects present in the input image and do not correspond to features or objects present in the golden sample image. The feature map analysis 826 may include identifying and locating features in the input image 802. Such features may include defects or the presence of particular parts. Such identification and location may include labeling and providing coordinates or bounding boxes for the identified features.

[0238] The identified features are provided to the centroid and shape analysis module 828 for centroid and shape analysis.

[0239] Referring now to Figure 13, shown therein is a method 1300 by which system 1200 operates, according to one embodiment. In Figure 13, steps similar to those in Figure 9 are indicated by the same last two reference numbers (e.g., 1302, 902).

[0240] Steps 1302 to 1308 function similarly to steps 902 to 908 in FIG. 9 and will not be repeated in detail here.

[0241] At 1320, a masked golden sample image is generated by applying a distinct mask to the golden sample image. This involves using the autoencoder output image to create a second (binary) image. The binary image is then used to mask the original input image. The golden sample image may be a generated golden sample image generated by providing the input image to a generative model trained to generate golden sample images.

[0242] At 1322, the masked input image 806 is provided to a first CNN 822 to generate an input image feature map.

[0243] At 1324, the masked golden sample image 812 is provided to a second CNN 824 to generate a golden sample feature map.

[0244] At 1326, a feature map analysis is performed on the input image feature map and the golden sample feature map to identify feature map differences. This feature map analysis may be in the form of a comparison.

[0245] Optionally, at 1328, centroid and shape analysis may be performed on the feature map differences identified at 1326. These operations include a combination of erosion and dilation operators on the binary image.

[0246] Referring now to FIG. 14, shown therein is a visual inspection processing pipeline 1400 for object detection image analysis with adaptive ROI segmentation according to one embodiment.

[0247] The pipeline 1400 begins with an input image 802, similar to FIG.

[0248] An input image 802 is sent to an adaptive ROI segmenter 804. The adaptive ROI segmenter 804 generates masked input image data 806.

[0249] The masked input image data 806 is provided as input to an object detection model 830. The object detection model 830 is configured to perform an object detection task on the masked input image data 806. The object detection model 830 is configured to locate and identify (assign a class label to) objects within the input image. Using the masked input image data 806 as input to the object detection model 830 advantageously promotes more efficient and effective computational processing, as the nature of the masked input image data 806 makes only the ROI available for processing by the object detection model 830. Thus, unnecessary processing and computation may be avoided.

[0250] The object detection model 830 generates object detection output data 832. The object detection output data 832 describes objects detected in the masked input image 806. The object detection output data 832 may include position data (e.g., bounding boxes) and a class label for each detected object. In particular embodiments, the detected objects may be defects or specific parts of the article being inspected.

[0251] Referring now to Figure 15, shown therein is a more detailed illustration of the object detection model 830 of Figure 14, according to one embodiment. The embodiment shown in Figure 15 includes a CNN architecture.

[0252] The object detection model 830 includes multiple components or layers. The components include an object proposal component 834, a feature map component 836, a region of interest (ROI) pooling component 838, a fully connected layer component 840, a softmax component 842, and a regressor component 844. The softmax component 842 provides a class label output for each object detected in the image 806. The regressor component 844 provides final bounding box coordinates for each object detected in the image 806. The bounding box coordinates define the target location (i.e., the detected object). The bounding box is a rectangular box that can be determined by the x-axis and y-axis coordinates of the upper left corner of the rectangle and the x-axis and y-axis coordinates of the lower right corner of the rectangle. The class label and final bounding box coordinate outputs may be combined into an object detection output 832 by the object detection model 830. The object detection output 832 may include a confidence score (e.g., between 0 and 1).

[0253] Referring now to Figure 16, shown therein is a method 1600 of visual inspection using adaptive ROI segmentation in the context of object detection image analysis, according to one embodiment. In Figure 16, steps 1602 through 1608 function similarly to steps 902 through 908 in Figure 9 and will not be described again here.

[0254] At 1630, the masked input image 806 is provided as an input to an object detection model 830 configured to detect at least one object class in the masked input image 802.

[0255] At 1632, object detection output data is generated, including detected object data for each object detected by the object detection model 830.

[0256] Although the above description provides one or more example devices, methods, or systems, one of ordinary skill in the art will recognize that other devices, methods, or systems are also within the scope of the claims.

Claims

1. 1. A system for visual inspection of an object using adaptive region of interest ("ROI") segmentation, comprising: a camera for acquiring an inspection image of the target item; an AI visual inspection computing device for detecting defects or abnormalities in the target article; The AI ​​visual inspection computing device includes: a communications interface for receiving the inspection image captured by the camera; an adaptive ROI segmentation module for processing an inspection image with an adaptive ROI segmentation model to generate a masked inspection image in which a region of non-interest ("nROI") is masked, the adaptive ROI segmentation limiting processing of the inspection image to a predetermined optimal area of ​​the inspection image; an image analysis module for receiving the mask inspection image and analyzing the mask inspection image with an image analysis model to generate output data indicative of the presence of the defects or anomalies detected by the image analysis model, wherein analysis of the mask inspection image is limited to an unmasked ROI; an output interface for displaying the output data; The image analysis module an object detection model configured to perform an object detection task on the mask test image and generate object detection output data describing one or more objects identified in the mask test image; a golden sample analysis module configured to generate a golden sample image of the target article and to generate golden sample output data describing a comparison between the mask inspection image and the golden sample image; a comparison module configured to compare the object detection output data generated by the object detection model with the golden sample output data generated by the golden sample analysis module, the comparing including comparing position data of the defects or anomalies detected in the object detection output data and in the golden sample output data. system.

2. 10. The system of claim 1, wherein the object detection model is trained to detect at least one defect class in the mask inspection image.

3. The system of claim 1 , wherein the golden sample analysis module includes a generative model for generating the golden sample image from the inspection image.

4. 3. The system of claim 2, wherein the output data includes a defect type and a defect location for each defect detected by the object detection model.

5. The system of claim 1 , wherein the adaptive ROI segmentation model is trained to identify and mask non-uniform areas of the inspection image.

6. 6. The system of claim 5, wherein the non-uniform areas include any one or more of improperly illuminated areas in the inspection image, user-defined non-uniform areas, components of the object that vary between different objects of the same class, and irregularly textured areas of the object.

7. The system of claim 2 , wherein the output data classifies the target item as either defective or non-defective.

8. 1. A method for visual inspection of an object using adaptive region of interest ("ROI") segmentation, comprising: acquiring an inspection image of the target item; processing the test image by masking n ROIs in the test image using an adaptive ROI segmentation model, wherein the processing of the test image is limited to a predetermined optimal area of ​​the test image by the adaptive ROI segmentation; analyzing the masked inspection image using an image analysis model to detect defects or anomalies in the target article; generating output data based on an output of the image analysis model, the output data indicating the presence of the detected defects or anomalies; displaying the output data on a user device; Including, The image analysis model is an object detection model configured to perform an object detection task on the mask test image and generate object detection output data describing one or more objects identified in the mask test image; a golden sample analysis module configured to generate a golden sample image of the target article and to generate golden sample output data describing a comparison between the mask inspection image and the golden sample image; a comparison module configured to compare the object detection output data generated by the object detection model with the golden sample output data generated by the golden sample analysis module, the comparing including comparing position data of the defects or anomalies detected in the object detection output data and in the golden sample output data. method.

9. The method of claim 8 , wherein the output data includes a defect type and a defect location for each defect detected by the object detection model.

10. The method of claim 8 , wherein the adaptive ROI segmentation model is trained to identify and mask non-uniform areas of the inspection image.

11. 11. The method of claim 10, wherein the non-uniform areas include one or more of improperly illuminated areas in the inspection image, user-defined non-uniform areas, components of the object that vary between different objects of the same class, and irregularly textured areas of the object.

12. 1. An AI visual inspection computing device for detecting objects in an inspection image using adaptive region of interest ("ROI") segmentation, comprising: a communication interface for receiving the inspection image captured by the camera; an adaptive ROI segmentation module for processing the inspection image with an adaptive ROI segmentation model to generate a masked inspection image in which a region of non-interest ("nROI") is masked, the adaptive ROI segmentation limiting processing of the inspection image to a predetermined optimal area of ​​the inspection image; and an image analysis module for receiving the mask inspection image and analyzing the mask inspection image with an image analysis model to generate output data indicative of the presence of the object detected by the image analysis model, wherein analysis of the mask inspection image is limited to an unmasked region of interest ("ROI"); an output interface for displaying the output data; The image analysis module an object detection model configured to perform an object detection task on the mask test image and generate object detection output data describing one or more objects identified in the mask test image; a golden sample analysis module configured to generate a golden sample image of the target article and to generate golden sample output data describing a comparison between the mask inspection image and the golden sample image; a comparison module configured to compare the object detection output data generated by the object detection model with the golden sample output data generated by the golden sample analysis module, the comparing including comparing location data of defects or anomalies detected in the object detection output data and in the golden sample output data. device.

13. The device of claim 12 , wherein the output data includes a defect type and a defect location for each defect detected by the object detection model.

14. The device of claim 12 , wherein the adaptive ROI segmentation model is trained to identify and mask non-uniform areas of the inspection image.

15. 15. The device of claim 14, wherein the non-uniform areas include one or more of improperly illuminated areas in the inspection image, user-defined non-uniform areas, components of the target article that vary between different target articles of the same class, and irregularly textured areas of the target article.

Images