Defect identification framework for manufacturing applications

Abstract

A computer-implemented method for defect identification based on a defect category driven image analysis. The method includes receiving an image of a product having a manufacturing process applied to the product; categorizing the image based on an attribute of the product in the image; selecting, based on the categorizing, a subset of image processing methods; initiating application of the selected subset of image processing methods to the image; receiving output images, each from the application of one of the selected subset of image processing methods; determining scores, each for one of the subset of the image processing methods, wherein each of the scores is based on metrics associated with different image qualities of a respective one of the output images; selecting an image processing method from the subset of imaging processing methods based on the scores; and identifying a defect in the product based on the selected imaging processing method.

Description

DEFECT IDENTIFICATION FRAMEWORK FOR MANUFACTURING APPLICATIONSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present application claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 511 ,885 entitled “Defect Identification Framework for Manufacturing Applications” and filed July 4, 2023, which is hereby incorporated herein by reference in its entirety as if fully set forth below and for all applicable purposes.STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

[0002] Not applicable.BACKGROUND

[0003] The manufacturing domain is an ever-growing field with different kinds of processes being studied and used in industrial settings. Some examples of manufacturing processes may include, but are not limited to, welding, casting, forming, semiconductors industry, machining (e.g., turning, milling, grinding), and additive manufacturing. Quality control in manufacturing is a systematic process that ensures products or product parts produced from manufacturing processes conform to a set of requirements (e.g., standards, customer expectations, performance, and reliability). To meet the ever-growing demand for high-quality products and address the disadvantages of manual quality control procedures, the use of intelligent visual inspection systems is gaining importance for deployment in production lines.SUMMARY

[0004] In an embodiment, a computer-implemented method for defect identification in manufacturing applications based on a defect category driven image analysis and a composite image quality scoring is disclosed. The method includes receiving, by a defect identification application stored in a non-transitory memory of a computer system and executed by a processor of the computer system, an image of a product having one or more manufacturing processes applied to the product; categorizing, by the defect identification application, the image into at least one of a plurality of manufacturing defect categories based on an attribute of the product in the image; selecting, by the defect identification application, based on the categorizing, a subset of image processing methods from a plurality of image processing methods to apply to the image;initiating, by the defect identification application, application of the selected subset of image processing methods to the image; receiving, by the defect identification application, a plurality of output images, each from the application of a respective one of the selected subset of image processing methods to the image; determining, by the defect identification application, a plurality of scores, each for a respective one of the subset of the image processing methods based on a respective one of the plurality of output images, wherein each of the plurality of scores is based on a combination of a plurality of metrics associated with different image qualities of a respective one of the output images; selecting, by the defect identification application, a recommended image processing method from the subset of imaging processing methods based on a comparison of the plurality of scores; and identifying, by the defect identification application, a defect in the product based on the recommended imaging processing method.

[0005] In another embodiment, a computer-implemented method for detecting defects in manufacturing applications based on a manufacturing process driven image analysis is disclosed. The method includes receiving, by a toolbox stored in a non-transitory memory of a computer system and executed by a processor of the computer system, an image of a product or product component; selecting, by the toolbox, a subset of image processing methods of a plurality of image processing methods to apply to the image based on a manufacturing process applied to the product or product component; initiating, by the toolbox, application of the selected subset of image processing methods; receiving, by the toolbox, results of the application of the selected subset of image processing methods; determining, by the toolbox, a recommendation score for each of the selected subset of image processing methods based on the results; selecting, by the toolbox, an image processing method of the subset of image processing methods based on the recommendation score associated with the image processing method; and providing, by the toolbox, the selected image processing method for use in defect detection for the product or product component.

[0006] In yet another embodiment, a system for defect identification is disclosed. The system includes a non-transitory memory; a processor; and a toolbox stored in the non- transitory memory that, when executed by the processor, causes the toolbox to receive an image of a product or product component; select a subset of image processing methods of a plurality of image processing methods to apply to the image based on at least one of a manufacturing process or a manufacturing defect category associatedwith the product or product component; initiate application of the selected subset of image processing methods; receive results of the application of the selected subset of image processing methods; determine a recommendation score for each of the selected subset of image processing methods based on the results; select an image processing method of the subset of image processing methods based on the recommendation score associated with the image processing method; and provide the selected image processing method for use in detecting a defect in the product or product component.

[0007] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0008] For a more complete understanding of the present disclosure, reference is now made to the following brief description, taken in connection with the accompanying drawings and detailed description, where like reference numerals represent like parts.

[0009] FIG. 1 is a block diagram of a manufacturing environment according to an embodiment of the disclosure.

[0010] FIG. 2 illustrates an example method of defect identification in manufacturing products or product components according to an embodiment of the disclosure.

[0011] FIG. 3 illustrates an example mapping of image processing methods to manufacturing defect categories according to an embodiment of the disclosure.

[0012] FIG. 4 is a table of example metrics and scores for image processing methods according to an embodiment of the disclosure.

[0013] FIG. 5 is a flow chart of a method according to an embodiment of the disclosure.

[0014] FIG. 6 is a flow chart of another method according to an embodiment of the disclosure.

[0015] FIG. 7 is a block diagram of a computer system according to an embodiment of the disclosure.DETAILED DESCRIPTION

[0016] It should be understood at the outset that although illustrative implementations of one or more embodiments are illustrated below, the disclosed systems and methods may be implemented using any number of techniques, whether currently known or not yet in existence. The disclosure should in no way be limited to the illustrative implementations, drawings, and techniques illustrated below, but may be modified within the scope of the appended claims along with their full scope of equivalents.

[0017] In a manufacturing setting, there are various types of defects. The type of defect encountered may depend on the type of manufacturing process that the product has undergone. Examples of defects may include ridges, pores, scratches, pits, cracks, or other types of surface defects. As imaging technologies advance, many of these imaging technologies are being considered for use in production, for example, at the end of production lines for post inspection and quality control. In some cases, imaging technologies may also be applied for on-line, intermittent, and in situ quality monitoring (e.g., at the output of every manufacturing process stage along a production line).

[0018] There are a variety of image processing techniques that are suitable for identifying or localizing defects (e.g., surface defects) in products or product components that are produced from manufacturing processes. However, each of these image processing techniques may be good at (or specialized in) detecting a particular type of defects. For example, a first image processing method may be good at detecting pores, a second image processing method may be good at detecting ridges, a third image processing method may be good at detecting scratches, a fourth image processing method may be good at detecting pits, a fifth image processing method may be good at detecting cracks, and a sixth image processing method may be good at detecting a different type of surface defect.

[0019] Recent advancements in machine learning (ML) also enabled the use of ML in identifying manufacturing defects. However, most of these ML methods are based on supervised learning. The success of supervised learning methods such as convolutional neural network (CNN) for defect detection is predicated on the availability of adequate labeled images. For instance, labeled images of the order of 104- 107samples are needed to adequately train these methods for defect detection in industrial scenarios. Considering the diversity of defects possible across various manufacturing chains, it may be time-consuming and costly to create a labeled image set that has a good coverage (including all types of defects) to be used for supervised learning methods. Further, in many industrial settings, a product or product part may undergo a sequence of manufacturing processes that require inspections after each process, and each process may result in surfaces of varying topographical features thereby different types of defects. Accordingly, it may be challenging to determine (or judge) which image processing techniques or models are to be used for what process in a complete manufacturing chain.

[0020] The present disclosure provides a technical solution to the aforementioned technical problems in the technical field of manufacturing defect identification to provide efficient and effective techniques for identifying and localizing defects in manufacturing applications. The present disclosure provides a manufacturing defect identification framework that utilizes a three-layer mapping between different types of defects from various manufacturing processes, image attributes of these defects, and different image processing methods. More specifically, the framework categorizes defects (e.g., surface defects) from various manufacturing processes into categories based on visual attributes (e.g., including dark lines, dark spots, periodic texture, and noisy background) of the defects. The framework identifies a pool of image processing methods (e.g., including image analysis techniques and ML techniques) suitable for identifying defects on surfaces of products or product components that have undergone various manufacturing processes. The framework maps each manufacturing defect category to a subset of the image processing methods (e.g., less than all of the image processing methods in the pool of image processing methods) that is most suitable or provides the best result for identifying defects of that particular manufacturing defect category. The mapping may be used to assist defect identification at runtime.

[0021] According to an embodiment of the present disclosure, a system for defect identification in manufacturing may include a defect identification toolbox application. The system may include a server or a computer including at least one non-transitory memory and at least one processor. The defect identification toolbox application may include instructions stored in the memory and executable by the processor. The defect identification toolbox application may be used for defect inspection and identification and may be applied across different manufacturing processes (e.g., welding, casting, forming, semiconductors industry, machining, additive manufacturing, etc.) and different categories of defects (e.g., dark lines, dark spots, periodic texture, noisy background, etc.).

[0022] The terms “toolbox” and “defect identification toolbox application” may be used interchangeably herein, such that a description referring to one of the terms shall be treated as though the description also referred to the other term.

[0023] In an embodiment, the defect identification toolbox application may receive an image of a product or product component that has undergone one or more manufacturing processes. For instance, the image may be captured by sensor(s) on a production floor. The defect identification toolbox application may categorize the imageinto one of the manufacturing defect categories (e.g., a dark line category, a dark spot category, a periodic texture category, and a noisy background category) based on a visual attribute of a surface of the product or product component captured in the image. Based on the categorization, the defect identification toolbox application may select a subset of image processing methods from a pool of image processing methods to apply to the image. The subset of image processing methods is less than all of the image processing methods in the pool. In an example, the pool of image processing methods may include a filtering and thresholding method, a basic line detector method, various deep learning methods, a Fourier-based analysis method, a wavelet-based analysis method, and / or a morphological component analysis (MCA) method (e.g., based on a directional pattern in an image).

[0024] The defect identification toolbox application may initiate application of the selected subset of image processing methods to the image. That is, the defect identification toolbox application may run the image through each of the image processing methods in the selected subset of image processing methods. In some examples, the subset of imaging processing methods may be implemented on another system, and the defect identification toolbox application may issue a command to that system to initiate the application of the subset of image processing methods. In other examples, the subset of image processing methods may be implemented on the same system as the defect identification toolbox application, and the defect identification toolbox application may call a function to initiate the application of the subset of image processing methods. Subsequently, the defect identification toolbox application may receive output images (e.g., segmented results or segmented images) from the application of the selected subset of image processing methods to the image. For example, each of the subset of image processing methods may output a corresponding one of the output images.

[0025] To evaluate the performance of the subset of image processing methods in detecting and / or localizing defects, the defect identification toolbox application may determine a score for each of the subset of the image processing methods based on a corresponding one of the output images. The defect identification toolbox application may compute each score based on a combination of metrics, each associated with a measure of a different image quality of a respective one of the output images. In an example, the metrics may include a first metric, a second metric, and a third metric. The first metric (e.g., represented by M1 ) may measure an entropy (e.g., the uniformity ofpixels) of a corresponding one of the output image. The second metric (e.g., represented by M2) may measure an average difference between intra-region disparity and interregion disparity in the corresponding output image. The third metric (e.g., represented by M3) may measure a loss of correlation, a luminance distortion, and / or a contrast distortion of the corresponding output image. In an example, the defect identification toolbox application may compute the score for an individual one of the subset of image processing methods based on a weighted sum of corresponding metrics (e.g., score = axM1 + bxM2 + cxM3, where the factors a, b, c may be any suitable values). In an embodiment, the weights or factors a, b, c for the first, second, and third metrics may respectively be -1 / 4, 1 / 4, and 1 / 2.

[0026] After determining a score for each of the subset of image processing methods, the defect identification toolbox application may select a recommended image processing method from the subset of imaging processing methods based on a comparison of the scores. For instance, the defect identification toolbox application may select the recommended image processing method based on the recommended image processing method having the highest score among the subset of image processing methods. Subsequently, the defect identification toolbox application may identify a defect (e.g., a pit, a ridge, a scratch, a crack, or a pore) in the product or product component based on the recommended imaging processing method. The defect identification toolbox application may also provide the selected image processing method for use in defect detection for other products or product components that may have defects under the same manufacturing defect category.

[0027] In an embodiment, the defect identification toolbox application may implement one or more ML models. For instance, the categorization of the image and the selection of the subset of image processing methods from the pool of image processing methods may be based on the one or more ML models. The ML models may be built based on categorizing different surface defects encountered across different manufacturing processes and mapping one or more particular image processing methods or image analysis tools to each category of defects.

[0028] As products or product components progress through the manufacturing processes, changes may be made to the manufacturing processes. As such, the defect identification toolbox application may periodically rerun its process on updated image(s) of products or product components to confirm that the previously recommended image processing method is still the recommended image processing method for the updatedproduct or product component. Otherwise, the defect identification toolbox application may select a different one of the image processing methods as the recommended image processing method for the updated product or product component.

[0029] In some cases, the manufacturing process that was applied to a certain product or product component is known. For instance, an image of the product or product component may be captured at the output of the manufacturing process. This may be the case when a computer system executing a defect identification toolbox application (e.g., a toolbox) is used for in situ quality monitoring (e.g., in a production line). Accordingly, in another embodiment of the present disclosure, a toolbox (including instructions) stored in a non-transitory memory of a computer system and executed by a processor of the computer system may receive an image of a product or product component. The toolbox may select a subset of image processing methods image processing methods to apply to the image based on a manufacturing process applied to the product or product component. In an example, the manufacturing process may cause a certain type of defects, which may be of a certain manufacturing defect category, and thus the mapping between manufacturing defect categories and image processing methods discussed above may be used to facilitate the selection of the subset of image processing methods. In other examples, the selection of the subset of image processing methods may be based on feedback from previously selected image processing methods for the same manufacturing process. The toolbox may initiate application of the selected subset of image processing methods. The toolbox may receive results of the application of the selected subset of image processing methods (e.g., in the form of segmented images). The toolbox may determine a recommendation score (e.g., a weighted sum of metrics) for each of the selected subset of image processing methods based on the results. The toolbox may select an image processing method of the subset of image processing methods based on the recommendation score associated with the image processing method. The toolbox may provide the selected image processing method for use in defect detection for the product or product component. The toolbox may also provide the selected image processing method for use in defect detection for other products or product components that have undergone the same manufacturing process.

[0030] The mapping and layered approach, which only applies a subset of image processing methods based on a manufacturing defect category or a manufacturing process instead of a complete set of image processing methods, results in more efficientprocessing of defects with less computer processing burden. The more efficient processing enables deployment of the defect identification toolbox application or toolbox in a production floor for real-time (or in situ) quality control monitoring. Using the defect identification toolbox application for real-time quality control monitoring can eliminate or reduce manual quality control that may be time inefficient and prone to human errors, and thus may improve the efficiency and effectiveness of defect identification. Further, using a composite image quality scoring evaluation (e.g., a certain combination of multiple different image quality metrics with a certain set of weights) to evaluate the subset of image processing methods can provide higher quality image processing method selection, resulting in more effective and accurate defect detection and identification.

[0031] Turning now to FIG. 1 , a manufacturing environment 100 is described. The manufacturing environment 100 includes a production line 110, a network 120, and a computer system 130. The production line 110 may include a chain of various manufacturing processes 112 (individually shown as 112-1 , 112-2, 112-3, ...). Some examples of manufacturing processes may include, but are not limited to, welding, casting, forming, semiconductors industry, machining, and additive manufacturing. A product or product part may generally undergo multiple different manufacturing processes. In the illustrated example of FIG. 1 , the manufacturing process 112-1 produces a product or product component 102-1 , which may further undergo the manufacturing process 112-2. The product or product component 102-2 may further undergo the manufacturing process 112-3 to produce a product or product component 102-3, and so on.

[0032] To assist quality inspection and control, the production line 110 may further include a plurality of sensors 114. The sensors 114 may be installed or located on the production floor at various outputs of the manufacturing processes 112. The sensors 114 may include image sensors for capturing images 118 of products or product components 102 output by the respective manufacturing processes 112. The sensors 114 may capture images of the products or product components 102 at different scales and / or at different resolutions. In some examples, the sensors 114 may include chromatic confocal based cameras. The production line 110 may further include one or more storage devices 116 to store the images 118 of the product or product components 102.

[0033] The network 120 promotes communication between the components of the manufacturing environment 100. The network 120 may be any communication network including a public data network (PDN), a public switched telephone network (PSTN), a private network, and / or a combination.

[0034] As discussed above, different types of manufacturing process 112 may cause different types of surface defects 104 (shown by the symbols “X”) in the product or product components 102. For instance, a welding manufacturing process 112 can cause undercuts and / or cracks in the product or product components 102. A forming manufacturing process 112 can cause wrinkles, tears, and / or imprints in the product or product components 102. A machining manufacturing process 112 can cause burrs, groove, chatter marks, deep lines, patches, adhered material, and / ortears in the product or product components 102. A casting manufacturing process 112 can cause defects 104 such as gas porosity, cavity, dendritic, shrinkage, and / or sponge in the product or product components 102. A semiconductor manufacturing process 112 may cause defects 104 such as dark lines, swirl, dark spots, stacking fault in the product or product components 102. An additive manufacturing process 112 can cause defects 104 such as micropores, lack of fusion, poor resin infusion, fiber pullout, surface roughness, and / or porosity in the product or product components 102. In general, defects 104 from manufacturing process 112 can be diverse, for example, in a variety of forms and shapes and at various locations of products or product components 102. As discussed above, an imaging processing method 136 may be effective in detecting a particular defect type but not all defect types. Further, in many industrial settings, a product or product part may undergo a sequence of manufacturing processes 112 that require inspections after each process 112, and each process 112 may result in surfaces of varying topographical features thereby different types of defects 104. Accordingly, it may be challenging to determine (or judge) which image processing techniques or models are to be used for what process in a complete manufacturing chain.

[0035] The computer system 130 may implement defect identification for manufacturing applications. The computer system 130 may include a defect identification toolbox application 132 including instructions stored in a non-transitory transitory memory of the computer system 130 and executable by one or more processors of the computer system 130. The computer system 130 may also include an image processing application 138 implementing a pool of image processing methods 136 (individually shown as 136-1 , 136-2, 136-3, .... 136-L, where L may be any suitable integer). In anembodiment, the image processing methods 136 may include a filtering and thresholding method, a basic line detector method, various deep learning methods, a Fourier-based analysis method, a wavelet-based analysis method, and / or an MCA method. The filtering and thresholding method may detect defects by removing noise and enhancing the image based on relevant pixel intensity. The basic line detector uses varying window sizes to capture small contrast differences to detect defects like scratches and micro-scratches. The deep learning methods may include CNNs (which are based on supervised learning) and semi-supervised methods such as generative adversarial networks (GANs), autoencoders, etc. The Fourier-based method models periodic background textures and utilizes the angular Fourier power spectrum to model the directionality of the surface for defect detection. The wavelet-based method analyzes signal components of different frequency components at different resolutions to extract out darker or lighter spots of various intensities and shapes from textured backgrounds for defect detection. The MCA method decomposes an image with respect to different texture descriptors like coarseness, directionality, line-likeness, etc. In a certain example, the MCA method analyzes the directional pattern on the surface of a product or product component 102 in an image 118 to identify a defect.

[0036] The defect identification toolbox application 132 may receive images 118 of the products or product components 102 that have undergone various types of manufacturing processes 112. As will be discussed more fully below with reference to FIGS. 2-6, the defect identification toolbox application 132 may utilize a defect category driven and / or manufacturing process 112 driven image analysis and a composite image quality scoring for detecting and identifying defects 104 in products or product components 102 output by the manufacturing processes 112. In some embodiments, the defect identification toolbox application 132 may perform the defect category driven and / or manufacturing process 112 driven image analysis using one or more ML models 134. The ML models 134 may be built based on categorizing different surface defects 104 encountered across different manufacturing processes 112 and mapping one or more particular image processing methods 136 to each category of defects 104.

[0037] In some embodiments, the computer system 130 may further include a user interface (Ul) 133 to provide various interfaces for a user to interact with the defect identification toolbox application 132. In an example, a user may select image processing method(s) 136 to be included or excluded for image analysis via the Ul 133. In another example, a user may change the default settings (e.g., default values forcertain image processing parameters) for certain image processing method(s) 136 via the III 133. In yet another example, the III 133 may provide a visual display of segmented results output by the image processing method(s) 136 (e.g., as shown in FIG. 2) and / or corresponding scores (e.g., as shown in FIG. 4).

[0038] FIG. 1 is merely an example of components of a manufacturing environment, and variations are contemplated to be within the scope of the present disclosure. In embodiments, the manufacturing environment may include other components not illustrated in FIG. 1. In embodiments, the manufacturing environment may not include every component illustrated in FIG. 1. In embodiments, the components and connections may be implemented with different connections than those illustrated in FIG. 1. Further, while FIG. 1 illustrates the image processing application 138 implemented as a separate application from the defect identification toolbox application 132 on the computer system 130, the image processing application 138 can be implemented on a separate computer system. Alternatively, the image processing application 138 can be implemented as part of the defect identification toolbox application 132. Further, in some cases, the images 118 of the products or product components 102 may be captured offline instead of using the sensors 114 (installed at the production line 110) as shown in FIG. 1. Such and other embodiments are contemplated to be within the scope of the present disclosure.

[0039] Turning now to FIG. 2, an example method 200 of defect identification in manufacturing products or product components is described. The method 200 may be implemented by the computer system 130, or more specifically the defect identification toolbox application 132 and the image processing application 138. For instance, the defect identification toolbox application 132 may receive an image 118 of a product or product component 102 having one or more manufacturing processes 112 applied to the product or product component 102. For instance, the image 118 may be captured by a sensor 114 on a production floor of the production line 110. At block 202, the defect identification toolbox application 132 may perform defect categorization. For instance, the defect identification toolbox application 132 may categorize the received image 118 into one of a plurality of manufacturing defect categories (e.g, the manufacturing defect categories 310, 312, 314, and 316 of FIG. 3) based on an attribute of a surface of the product or product component 102 in the image 118.

[0040] At block 204, the defect identification toolbox application 132 may perform image processing methods 136 selection. For instance, based on the defect categorization atblock 202, the defect identification toolbox application 132 may select a subset of image processing methods 136 from the pool of image processing methods 136 (implemented by the image processing application 138) to apply to the image 118. The subset of image processing methods 136 is less than all of the image processing methods 136 in the image processing application 138. The subset of image processing methods 136 may be based on a mapping of image processing methods to manufacturing defect categories as will be discussed more fully below with reference to FIG. 3.

[0041] After selecting the subset of image processing methods 136, the defect identification toolbox application 132 may initiate application of the selected subset of image processing methods 136 to the image 118. That is, the defect identification toolbox application 132 may run the image 118 through each of the image processing methods 136 in the selected subset of image processing methods 136. In the illustrated example of FIG. 2, the selected subset of image processing methods 136 includes the image processing methods 136-1 , 136-3, and 136-L (shown by the patterned boxes). Stated differently, the defect identification toolbox application 132 excludes the image processing methods 136-2 and 136-4 to 136-(L-1 ) based on the manufacturing defect category associated with the image 118. In an example, the defect identification toolbox application 132 may initiate the application of the subset of image processing methods 136 to the image 118 by calling a function or application programming interface (API) provided by the image processing application 138. In other examples, when the subset of imaging processing methods 136-1 , 136-3, and 136-L is implemented on a different computer system than the defect identification toolbox application 132, the defect identification toolbox application 132 may issue a command to that system to initiate the application of the subset of image processing methods 136-1 , 136-3, and 136-L to the image 118.

[0042] Turning now to FIG. 3, an example mapping 300 of image processing methods 136 to manufacturing defect categories 310, 312, 314, and 316 is described. The categories 310, 312, 314, and 316 may be determined based on certain visual attributes of images 118 of products or product components 102 (output by various manufacturing process 112). For instance, defects 104 that manifest in any manufacturing process 112 can be categorized based on certain image attributes namely, smooth background with dark lines and dark spots, periodic textured background, and noisy background. Accordingly, a plurality of manufacturing defect categories may include a dark linecategory 310, a dark spot category 312, a periodic texture category 314, and a noisy background category 316

[0043] As shown in FIG. 3, the images 118 (of the products or product components 102 that have undergone one or more manufacturing processes 112) may be categorized into at least one of the plurality of manufacturing defect categories 310, 312, 314, and 316. As an example, defects 104 in the form of cracks, surface roughness, dark line defects, deep lines (from grinding), roughness (from milling), and / or patches (from grinding) can be categorized under the dark line category 310. As another example, defects 104 in the form of micropores, fiber pullout, porosity, gas porosity, undercutting, shrinkage, shade (from grinding), tears, adhered material (from lathing), and / or patches (from grinding) can be categorized under the dark spot category 312. As yet another example, defects 104 in the form of tears, groove (from lathing), dark spot defects, poor resin infusion, porosity, swirl defects, roughness (from milling), and adhered material (from lathing) can be categorized under the periodic texture category 314. As a further example, defects 104 in the form of dark spot defects, micropores, fiber pullout, burrs, gas porosity, and shrinkage can be categorized under the noisy background category 316.

[0044] As further shown in FIG. 3, a subset of the image processing methods 136 are mapped to each of the manufacturing defect categories 310, 312, 314, and 316. For ease of illustration, FIG. 3 only shows a pool of 9 image processing methods 136-1 to 136-9 being mapped to the manufacturing defect categories 310, 312, 314, and 316. In general, the image processing application 138 can implement a pool of any suitable number of image processing methods 136, and the defect identification toolbox application 132 may map any suitable number of image processing methods 136 to each manufacturing defect category 310, 312, 314, and 316. Further, in some cases, the same image processing method 136 may be mapped to more than one manufacturing defect category 310-316. That is, there can be overlapping image processing method(s) 136 between two or more of the manufacturing defect categories 310-316. In the illustrated example of FIG. 3, the image processing methods 136-1 and 136-3 are mapped to the dark line category 310, the image processing methods 136-1 and 136-2 are mapped to the dark sport category 312, the image processing methods 136-4 to 136-7 are mapped to the periodic texture category 314, and the image processing methods 136-8 and 136-9 are mapped to the noisy background category 316.

[0045] In an embodiment, image processing methods 136 including filtering and threshold methods, basic line detector method, deep learning methods (e.g., CNNs, GANs, autoencoders, etc.) may be suitable for detecting defects 104 under the dark line category 310 and the dark spot category 312. Image processing methods 136 including Fourier-based analysis, wavelet-based analysis, MCA analysis, and deep learning methods (e.g., CNNs, GANs, autoencoders, etc.) may be suitable for detecting defects 104 under the periodic structure category 314. Image processing methods 136 including various deep learning methods (e.g, CNNs, GANs, autoencoders, etc.) may be suitable for detecting defects 104 under the noisy background category 316. In an embodiment, the deep learning methods for the different manufacturing defect categories 310-316 may include different ML models trained and / or designed specifically for detecting the respective types of defects 104 or manufacturing defect category 310, 312, 314, or 316.

[0046] Generally, a subset of the image processing methods 136 from the pool of image processing methods 136 can be mapped to each manufacturing defect category 310, 312, 314, or 316. Because certain types of manufacturing process 112 may produce certain types of defects, a subset of the image processing methods 136 from the pool of image processing methods 136 can also be mapped to a certain type of manufacturing process 112. The mapping of a subset of image processing methods 136 to each manufacturing defect category 310, 312, 314, or 316 or to each manufacturing process 112 can greatly reduce the computational intensity and complexity compared to having to apply each of the complete set of image processing methods 136 (in the pool) to an image 118.

[0047] Returning to FIG. 2, at block 206, based on the initiation from the defect identification toolbox application 132 at block 204, the image processing application 138 may apply the selected subset of image processing methods 136-1 , 136-3, and 136-L to the image 118. That is, the image 118 may be provided as an input to each of the subset of image processing methods 136-1 , 136-3, and 136-L, and each of subset of image processing methods 136-1 , 136-3, and 136-L may process the image 118. Each of the subset of image processing methods 136-1 , 136-3, and 136-L may generate an output image 208 based on the processing of the image 118. As shown, the image processing method 136-1 outputs an output image 208-1 , the image processing method 136-3 outputs an output image 208-2, the image processing method 136-L outputs an output image 208-3. The output images 208 may include a segmented result (or segmented image) from processing the input image 118. A segmented image 208 mayinclude a set of segments extracted from the image 118 that collectively cover the entire image 118 or a set of contours extracted from the image 118.

[0048] In the illustrated example of FIG. 2, the image processing method 136-1 may be a base line detector, the image processing method 136-3 may be a wavelet-based analysis method, and the image processing method 136-1 may be an MCA method. As can be observed, each of the image processing methods 136-1 , 136-3, and 136-L provides a different segmented result (the output images 208). These output images 208 may be provided to the defect identification toolbox application 132 for evaluation.

[0049] For instance, at block 210, the defect identification toolbox application 132 may receive the output images 208 and may perform score determination. For instance, the defect identification toolbox application 132 may determine a score (e.g., the scores 404 shown in FIG. 4) for each of the selected subset of the image processing methods 136- 1 , 136-3, and 136-L based on a corresponding one of the output images 208. The defect identification toolbox application 132 may compute each score based on a combination of metrics (e.g., the metrics 402 shown in FIG. 4). Each metric may be associated with a measure of a different image quality of a corresponding one of the output images 208. For instance, the defect identification toolbox application 132 may compute a first metric, a second metric, and a third metric for each of the output images 208. In some embodiments, the defect identification toolbox application 132 may compute more or less than three metrics. The selection of the combination of the metrics may impact the effectiveness of the evaluation. In an embodiment, the first metric (e.g., represented by M1) may measure an entropy of a corresponding one of the output image 208. The second metric (e.g., represented by M2) may measure an average difference between intra-region disparity and inter-region disparity in the corresponding output image 208. The second metric may be referred to as a Rosenberger’s metric. The third metric (e.g., represented by M3) may measure a loss of correlation, a luminance distortion, and / or a contrast distortion of the corresponding output image 208. The third metric may be referred to as a universal quality index (UQI). The defect identification toolbox application 132 may compute the score for each of the subset of image processing methods 136-1 , 136-3, and 136-L based on a weighted sum of corresponding metrics (e.g., score = a*M1 + b*M2 + cxM3, where the factors a, b, c may be any suitable values). The selection of the weights or factors a, b, c may impact the effectiveness of the evaluation. In an embodiment, the weights or factors a, b, c for the first, second, and third metrics may respectively be -1 / 4, 1 / 2, and 1 / 4. That is, a score for an output image208 with a Rosenberger’s metric (e.g., the second metric), a UQI metric (e.g., the third metric), and an entropy metric (e.g., the first metric) may be computed as shown below:In some embodiments, different weights and / or different metrics may be used to produce the score for each image without departing from the spirit or scope of the disclosure.

[0050] Turning now to FIG. 4, a table 400 of example metrics and scores for the selected subset of image processing methods 136-1 , 136-3, and 136-L is described. The table 400 shows the metrics 402 (e.g., an entropy metric, a UQI metric, and a Rosenberger’s metric) computed for each of the selected subset of image processing methods 136-1 , 136-3, and 136-L. The table 400 further shows a score 404 computed for each of the selected subset of image processing methods 136-1 , 136-3, and 136-L based on a weighted sum of respective metrics 402 discussed above.

[0051] Returning to FIG. 2, at block 212, after determining a score 404 for each of the subset of image processing methods 136-1 , 136-3, and 136-L, the defect identification toolbox application 132 may perform recommended image processing method selection. For instance, the defect identification toolbox application 132 may select a recommended image processing method, e.g., 136-L, from the subset of imaging processing methods 136-1 , 136-3, and 136-L based on a comparison of the scores 404. For instance, the defect identification toolbox application 132 may select the recommended image processing method 136 based on the recommended image processing method 136 having the highest score 404 among the subset of image processing methods 136-1 , 136-3, and 136-L.

[0052] At block 214, after selecting the recommend image processing method 136-L, the defect identification toolbox application 132 may identify a defect 104 in the product or product component 102 in the image 118 based on the recommended imaging processing method 136-L. For instance, the defect identification toolbox application 132 may identify the defect 104 from the output image 208-3 generated by the recommended image processing method 136-L.

[0053] In an embodiment, the defect identification toolbox application 132 may implement one or more ML models 134, and the defect categorization at block 202 and the image processing methods selection at block 204 may be performed using the one or more ML models 134 (e.g., by feeding the image 118 into the one or more ML models134 for processing by the one or more ML models 134). The ML models 134 may be built based on categorizing different surface defects 104 encountered across different manufacturing processes 112 and mapping one or more particular image processing methods 136 to each manufacturing defect category 310-316. In some examples, the ML models 134 may include CNN(s). In general, the ML models 134 may have any suitable ML architecture.

[0054] In an example, to train an ML model 134, a set of images 118 of products or product components 102 (that have undergone various manufacturing processesl 12) may be labeled to create a set of labelled images. For instance, each image 118 may be labelled with a most suitable subset of image processing methods 136 and / or a recommended image processing method 136 selected from the subset of image processing method 136. The set of labelled images may be input to an ML model 134. The ML model 134 may include activation functions and corresponding parameters. The ML model 134 may process each labelled image and output a list of image processing methods 136 and an indication of a recommended image processing method 136 out of the list of image processing methods 136. The list of image processing methods 136 may correspond to the subset of image processing methods 136 selected from a complete set of image processing methods 136 (e.g., based on a manufacturing defect category associated with the labelled image discussed above). The recommended image processing method 136 may correspond to the image processing method 136 selected from the subset of image processing methods 136 (e.g., using the scoring method discussed above with reference to equation (1)). In other examples, the ML model 136 may output a list of image processing methods 136 and a ranking for the image processing methods 136, where the image processing method 136 that ranked the highest may correspond to the recommended image processing method 136. The output of the ML model 134 may be evaluated, and the ML model 134 may be updated based on the evaluation. For instance, the evaluation may include comparing the list of image processing methods 136 and / or the recommended image processing method 136 output by the ML model 134 to the ground truth provided by the labels (e.g., the labels indicating the subset of image processing methods 136 and the recommended image processing method 136 for the corresponding input training image). The evaluation may further include determining an error measure for the ML model 134 based on the comparison. The ML model 134 (e.g., the parameters of the ML model 134) may be updated based on the evaluation result. The training process of feedingthe ML model 134 with a labelled image, evaluating the output of the ML model 134 against the ground truth, and updating the ML model based on the evaluation may be repeated until an error measure from the evaluation is satisfied (e.g., satisfying a threshold). In some examples, the set of labelled images used for the training may include image samples of products or product components 102 with defects 104 from various manufacturing defect categories 310-316 and / or products or product components 102 produced from various manufacturing processes 112.

[0055] As products or product components 102 progress through the manufacturing processes 112, changes may be made to the manufacturing processes 112. As such, the defect identification toolbox application 132 may periodically rerun its process (e.g., the operations at blocks 202, 204, 206, 210, and 212 of the method 200) on updated image(s) 118 of products or product components 102 to confirm that the previously recommended image processing method 136 is still the recommended image processing method 136 for the updated product or product component 102. Otherwise, the defect identification toolbox application 132 may select a different one of the image processing methods 136 as the recommended image processing method for the updated product or product component 102.

[0056] The method 200 may be applied in various scenarios for defect identification in manufacturing. In one embodiment, the method 200 may be implemented as an offline defect identification tool (e.g., at the end of the production line 110). For instance, the sensors 114 may capture images 118 of the products or product components 102 produced by the manufacturing processes 112 and store the captured images 118 in the storage device(s) 116 for defect identification. The defect identification toolbox application 132 may retrieve the stored images 118 (e.g., via the network 120) and apply the method 200 for post analysis. In another embodiment, the method 200 may be implemented as a real-time quality control monitoring tool. In such an embodiment, the defect identification toolbox application 132 may receive an image 118 directly from a sensor 114 or from the storage device(s) 116 (via the network 120) as the image 118 is captured at the output of a corresponding manufacturing process 112 and may apply the method 200 for real-time quality control monitoring. The defect identification toolbox application 132 may raise an alert upon detecting a defect in a product or product component 102 captured by the received image 118. Because of the layered approach in which only a subset of image processing methods 136 is selected for processing animage 118, the computational complexity can be greatly reduced, enabling the method 200 to be performed in real-time or close to real-time.

[0057] Turning now to FIG. 5, a method 500 is described. In an embodiment, the method 500 is a method for performing a defect category driven image analysis for defect identification in manufacturing applications. The method 500 may include similar mechanisms as discussed above with reference to FIGS. 1-4. In embodiments, the method 500 may be implemented using a computer system with components as shown in FIG. 7. The method 500 may be implemented by a defect identification toolbox application 132 including instructions stored in a non-transitory memory of a computer system 130 and executable by a processor of the computer system 130. As illustrated, FIG. 5 includes a number of enumerated operations, but embodiments of the operations in FIG. 5 may include additional operations before, after, and in between the enumerated operations. In some embodiments, one or more of the enumerated operations may be omitted or performed in a different order.

[0058] At block 502, the defect identification toolbox application 132 receives an image 118 of a product 102 (e.g., a complete product, a product part, or a product component) having one or more manufacturing processes 112 applied to the product 102.

[0059] At block 504, the defect identification toolbox application 132 categorizes the image 118 into at least one of a plurality of manufacturing defect categories 310-316 based on an attribute (e.g., a visual service attribute) of the product 102 in the image 118. In an embodiment, the plurality of manufacturing defect categories comprises at least one of a dark line category 310, a dark spot category 312, a periodic structure category 314, or a noisy background category 316.

[0060] At block 506, the defect identification toolbox application 132 selects, based on the categorizing at block 504, a subset of image processing methods 136 from a plurality of image processing methods 136 to apply to the image 118. In an embodiment, the plurality of imaging processing methods 136 comprises at least one of a filtering and thresholding method, a basic line detector method, one or more deep learning methods, a Fourier-based analysis method, a wavelet-based analysis method, or an MCA method. In an embodiment, the MCA method is based on a directional pattern on a surface of the image 118.

[0061] At block 508, the defect identification toolbox application 132 initiates application of the selected subset of image processing methods 136 to the image 118.

[0062] At block 510, the defect identification toolbox application 132 receives a plurality of output images 208, each from the application of a respective one of the selected subset of image processing methods 136 to the image 118.

[0063] At block 512, the defect identification toolbox application 132 determines a plurality of scores 404, each for a respective one of the subset of the image processing methods 136 based on a respective one of the plurality of output images 208. Each of the plurality of scores 404 is based on a combination of a plurality of metrics 402 associated with different image qualities of a respective one of the output images 208. In an embodiment, as part of determining the plurality of scores 404, the defect identification toolbox application 132 determines an individual score 404 of the plurality of scores 404 for a respective one of the subset of the image processing methods 136 based on a weighted sum of the plurality of metrics 402. In an embodiment, the plurality of metrics 402 used for determining each of the plurality of scores includes a first metric 402 based on an entropy measure of the image 118, a second metric 402 based on an average difference between intra-region disparity and inter-region disparity in the image 118, and a third metric 402 based on at least one of a loss of correlation, a luminance distortion, or a contrast distortion in the image 118.

[0064] At block 514, the defect identification toolbox application 132 selects a recommended image processing method 136 from the subset of imaging processing methods 136 based on a comparison of the plurality of scores 404.

[0065] At block 516, the defect identification toolbox application 132 identifies a defect 104 in the product 102 based on the recommended imaging processing method 136. In an embodiment, the defect 104 identified based on the recommended image processing method 136 comprises at least one of a pit, a ridge, a scratch, a crack, or a pore.

[0066] In an embodiment, at least one of the categorizing the image 118 at block 504 or the selecting the subset of the image processing methods 136 at block 506 are based on ML.

[0067] Turning now to FIG. 6, a method 600 is described. In an embodiment, the method 600 is a method for performing a manufacturing process driven image analysis for defect identification in manufacturing applications. The method 600 may include similar mechanisms as discussed above with reference to FIGS. 1-5. The method 600 may be implemented by a toolbox (e.g., the defect identification toolbox application 132) including instructions stored in a non-transitory memory of a computer system 130 and executable by a processor of the computer system 130. In embodiments, the method600 may be implemented using a computer system with components as shown in FIG.7. As illustrated, FIG. 6 includes a number of enumerated operations, but embodiments of the operations in FIG. 6 may include additional operations before, after, and in between the enumerated operations. In some embodiments, one or more of the enumerated operations may be omitted or performed in a different order.

[0068] At block 602, the toolbox (e.g., the defect identification toolbox application 132) receives an image 118 of a product or product component 102.

[0069] At block 604, the toolbox selects a subset of image processing methods 136 of a plurality of image processing methods 136 to apply to the image 118 based on a manufacturing process 112 applied to the product or product component 102.

[0070] At block 606, the toolbox initiates application of the selected subset of image processing methods 136.

[0071] At block 608, the toolbox receives results (e.g., segmentation results, the output images 208) of the application of the selected subset of image processing methods 136.

[0072] At block 610, the toolbox determines a recommendation score 404 for each of the selected subset of image processing methods 136 based on the results.

[0073] At block 612, the toolbox selects an image processing method 136 of the subset of image processing methods 136 based on the recommendation score 404 associated with the image processing method 136.

[0074] At block 614, the toolbox provides the selected image processing method 136 for use in defect detection for the product or product component 102.

[0075] In an embodiment, the toolbox further applies the selected image processing method 136 to one or more images 118 of the product or product component 102.

[0076] In an embodiment, the toolbox further initiates application of the selected image processing method to one or more images 118 of the product or product component 102. In an embodiment, as part of initiating the selected image processing method 136, the toolbox issues a command to a corresponding system that applies the selected image processing method 136.

[0077] In an embodiment, the toolbox further receives an updated image 118 of the product or product component 102. The toolbox selects a second subset of image processing methods 136 of the plurality of image processing methods 136 to apply to the updated image 118 based on a manufacturing process 112 applied to the product or product component 102. The toolbox initiates application of the selected second subset of image processing methods 136. The toolbox receives results of the applicationof the selected second subset of image processing methods 136. The toolbox determines a recommendation score 404 for each of the selected second subset of image processing methods 136 based on the results of application of the selected second subset of image processing methods 136. The toolbox selects a second image processing method 136 of the second subset of image processing methods 136 based on the recommendation score 404 associated with the second image processing method 136. The toolbox provides the selected second image processing method 136 for use in defect detection for the product or product component 102, where the selected image processing method 136 and the selected second image processing method 136 are different. In an embodiment, the subset of image processing methods 136 and the second subset of image processing methods 136 are the same.

[0078] In an embodiment, the toolbox implements one or more ML models 134. In an embodiment, the toolbox further tunes the one or more ML models 134 based on feedback about previously selected image processing methods 136 for images 118 of products or product components 102.

[0079] FIG. 7 illustrates a computer system 380 suitable for implementing one or more embodiments disclosed herein. The computer system 380 includes a processor 382 (which may be referred to as a central processor unit or CPU) that is in communication with memory devices including secondary storage 384, read only memory (ROM) 386, RAM 388, input / output (I / O) devices 390, and network connectivity devices 392. The processor 382 may be implemented as one or more CPU chips.

[0080] It is understood that by programming and / or loading executable instructions onto the computer system 380, at least one of the CPU 382, the RAM 388, and the ROM 386 are changed, transforming the computer system 380 in part into a particular machine or apparatus having the novel functionality taught by the present disclosure. It is fundamental to the electrical engineering and software engineering arts that functionality that can be implemented by loading executable software into a computer can be converted to a hardware implementation by well-known design rules. Decisions between implementing a concept in software versus hardware typically hinge on considerations of stability of the design and numbers of units to be produced rather than any issues involved in translating from the software domain to the hardware domain. Generally, a design that is still subject to frequent change may be preferred to be implemented in software, because re-spinning a hardware implementation is more expensive than re-spinning a software design. Generally, a design that is stable that willbe produced in large volume may be preferred to be implemented in hardware, for example in an application specific integrated circuit (ASIC), because for large production runs the hardware implementation may be less expensive than the software implementation. Often a design may be developed and tested in a software form and later transformed, by well-known design rules, to an equivalent hardware implementation in an ASIC that hardwires the instructions of the software. In the same manner as a machine controlled by a new ASIC is a particular machine or apparatus, likewise a computer that has been programmed and / or loaded with executable instructions may be viewed as a particular machine or apparatus.

[0081] Additionally, after the system 380 is turned on or booted, the CPU 382 may execute a computer program or application. For example, the CPU 382 may execute software or firmware stored in the ROM 386 or stored in the RAM 388. In some cases, on boot and / or when the application is initiated, the CPU 382 may copy the application or portions of the application from the secondary storage 384 to the RAM 388 or to memory space within the CPU 382 itself, and the CPU 382 may then execute instructions that the application is comprised of. In some cases, the CPU 382 may copy the application or portions of the application from memory accessed via the network connectivity devices 392 or via the I / O devices 390 to the RAM 388 or to memory space within the CPU 382, and the CPU 382 may then execute instructions that the application is comprised of. During execution, an application may load instructions into the CPU 382, for example load some of the instructions of the application into a cache of the CPU 382. In some contexts, an application that is executed may be said to configure the CPU 382 to do something, e.g., to configure the CPU 382 to perform the function or functions promoted by the subject application. When the CPU 382 is configured in this way by the application, the CPU 382 becomes a specific purpose computer or a specific purpose machine.

[0082] The secondary storage 384 is typically comprised of one or more disk drives or tape drives and is used for non-volatile storage of data and as an over-flow data storage device if RAM 388 is not large enough to hold all working data. Secondary storage 384 may be used to store programs which are loaded into RAM 388 when such programs are selected for execution. The ROM 386 is used to store instructions and perhaps data which are read during program execution. ROM 386 is a non-volatile memory device which typically has a small memory capacity relative to the larger memory capacity of secondary storage 384. The RAM 388 is used to store volatile data and perhaps to storeinstructions. Access to both ROM 386 and RAM 388 is typically faster than to secondary storage 384. The secondary storage 384, the RAM 388, and / or the ROM 386 may be referred to in some contexts as computer readable storage media and / or non-transitory computer readable media.

[0083] I / O devices 390 may include printers, video monitors, liquid crystal displays (LCDs), touch screen displays, keyboards, keypads, switches, dials, mice, track balls, voice recognizers, card readers, paper tape readers, or other well-known input devices.

[0084] The network connectivity devices 392 may take the form of modems, modem banks, Ethernet cards, USB interface cards, serial interfaces, token ring cards, fiber distributed data interface (FDDI) cards, wireless local area network (WLAN) cards, radio transceiver cards, and / or other well-known network devices. The network connectivity devices 392 may provide wired communication links and / or wireless communication links (e.g., a first network connectivity device 392 may provide a wired communication link and a second network connectivity device 392 may provide a wireless communication link). Wired communication links may be provided in accordance with Ethernet (IEEE 802.3), Internet protocol (IP), time division multiplex (TDM), data over cable service interface specification (DOCSIS), wavelength division multiplexing (WDM), and / or the like. In an embodiment, the radio transceiver cards may provide wireless communication links using protocols such as CDMA, global system for mobile communications (GSM), LTE, WiFi (IEEE 802.11), Bluetooth, Zigbee, narrowband Internet of things (NB loT), near field communications (NFC), and radio frequency identity (RFID). The radio transceiver cards may promote radio communications using 5G, 5G New Radio, or 5G LTE radio communication protocols. These network connectivity devices 392 may enable the processor 382 to communicate with the Internet or one or more intranets. With such a network connection, it is contemplated that the processor 382 might receive information from the network, or might output information to the network in the course of performing the above-described method steps. Such information, which is often represented as a sequence of instructions to be executed using processor 382, may be received from and outputted to the network, for example, in the form of a computer data signal embodied in a carrier wave.

[0085] Such information, which may include data or instructions to be executed using processor 382 for example, may be received from and outputted to the network, for example, in the form of a computer data baseband signal or signal embodied in a carrier wave. The baseband signal or signal embedded in the carrier wave, or other types ofsignals currently used or hereafter developed, may be generated according to several methods well-known to one skilled in the art. The baseband signal and / or signal embedded in the carrier wave may be referred to in some contexts as a transitory signal.

[0086] The processor 382 executes instructions, codes, computer programs, scripts which it accesses from hard disk, floppy disk, optical disk (these various disk-based systems may all be considered secondary storage 384), flash drive, ROM 386, RAM 388, or the network connectivity devices 392. While only one processor 382 is shown, multiple processors may be present. Thus, while instructions may be discussed as executed by a processor, the instructions may be executed simultaneously, serially, or otherwise executed by one or multiple processors. Instructions, codes, computer programs, scripts, and / or data that may be accessed from the secondary storage 384, for example, hard drives, floppy disks, optical disks, and / or other device, the ROM 386, and / or the RAM 388 may be referred to in some contexts as non-transitory instructions and / or non-transitory information.

[0087] In an embodiment, the computer system 380 may comprise two or more computers in communication with each other that collaborate to perform a task. For example, but not by way of limitation, an application may be partitioned in such a way as to permit concurrent and / or parallel processing of the instructions of the application. Alternatively, the data processed by the application may be partitioned in such a way as to permit concurrent and / or parallel processing of different portions of a data set by the two or more computers. In an embodiment, virtualization software may be employed by the computer system 380 to provide the functionality of a number of servers that is not directly bound to the number of computers in the computer system 380. For example, virtualization software may provide twenty virtual servers on four physical computers. In an embodiment, the functionality disclosed above may be provided by executing the application and / or applications in a cloud computing environment. Cloud computing may comprise providing computing services via a network connection using dynamically scalable computing resources. Cloud computing may be supported, at least in part, by virtualization software. A cloud computing environment may be established by an enterprise and / or may be hired on an as-needed basis from a third-party provider. Some cloud computing environments may comprise cloud computing resources owned and operated by the enterprise as well as cloud computing resources hired and / or leased from a third-party provider.

[0088] In an embodiment, some or all of the functionality disclosed above may be provided as a computer program product. The computer program product may comprise one or more computer readable storage medium having computer usable program code embodied therein to implement the functionality disclosed above. The computer program product may comprise data structures, executable instructions, and other computer usable program code. The computer program product may be embodied in removable computer storage media and / or non-removable computer storage media. The removable computer readable storage medium may comprise, without limitation, a paper tape, a magnetic tape, magnetic disk, an optical disk, a solid state memory chip, for example analog magnetic tape, compact disk read only memory (CD-ROM) disks, floppy disks, jump drives, digital cards, multimedia cards, and others. The computer program product may be suitable for loading, by the computer system 380, at least portions of the contents of the computer program product to the secondary storage 384, to the ROM 386, to the RAM 388, and / or to other non-volatile memory and volatile memory of the computer system 380. The processor 382 may process the executable instructions and / or data structures in part by directly accessing the computer program product, for example by reading from a CD-ROM disk inserted into a disk drive peripheral of the computer system 380. Alternatively, the processor 382 may process the executable instructions and / or data structures by remotely accessing the computer program product, for example by downloading the executable instructions and / or data structures from a remote server through the network connectivity devices 392. The computer program product may comprise instructions that promote the loading and / or copying of data, data structures, files, and / or executable instructions to the secondary storage 384, to the ROM 386, to the RAM 388, and / or to other non-volatile memory and volatile memory of the computer system 380.

[0089] In some contexts, the secondary storage 384, the ROM 386, and the RAM 388 may be referred to as a non-transitory computer readable medium or a computer readable storage media. A dynamic RAM embodiment of the RAM 388, likewise, may be referred to as a non-transitory computer readable medium in that while the dynamic RAM receives electrical power and is operated in accordance with its design, for example during a period of time during which the computer system 380 is turned on and operational, the dynamic RAM stores information that is written to it. Similarly, the processor 382 may comprise an internal RAM, an internal ROM, a cache memory, and / or other internal non-transitory storage blocks, sections, or components that maybe referred to in some contexts as non-transitory computer readable media or computer readable storage media.

[0090] While several embodiments have been provided in the present disclosure, it should be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples are to be considered as illustrative and not restrictive, and the intention is not to be limited to the details given herein. For example, the various elements or components may be combined or integrated in another system or certain features may be omitted or not implemented.

[0091] Also, techniques, systems, subsystems, and methods described and illustrated in the various embodiments as discrete or separate may be combined or integrated with other systems, modules, techniques, or methods without departing from the scope of the present disclosure. Other items shown or discussed as directly coupled or communicating with each other may be indirectly coupled or communicating through some interface, device, or intermediate component, whether electrically, mechanically, or otherwise. Other examples of changes, substitutions, and alterations are ascertainable by one skilled in the art and could be made without departing from the spirit and scope disclosed herein.

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method for defect identification in manufacturing applications based on a defect category driven image analysis and a composite image quality scoring evaluation, the method comprising: receiving, by a defect identification application stored in a non-transitory memory of a computer system and executed by a processor of the computer system, an image of a product having one or more manufacturing processes applied to the product; categorizing, by the defect identification application, the image into at least one of a plurality of manufacturing defect categories based on an attribute of the product in the image; selecting, by the defect identification application, based on the categorizing, a subset of image processing methods from a plurality of image processing methods to apply to the image; initiating, by the defect identification application, application of the selected subset of image processing methods to the image; receiving, by the defect identification application, a plurality of output images, each from the application of a respective one of the selected subset of image processing methods to the image; determining, by the defect identification application, a plurality of scores, each for a respective one of the subset of the image processing methods based on a respective one of the plurality of output images, wherein each of the plurality of scores is based on a combination of a plurality of metrics associated with different image qualities of a respective one of the output images; selecting, by the defect identification application, a recommended image processing method from the subset of imaging processing methods based on a comparison of the plurality of scores; and identifying, by the defect identification application, a defect in the product based on the recommended imaging processing method.

2. The method of claim 1 , wherein the plurality of manufacturing defect categories comprises at least one of: a dark line category, a dark spot category,a periodic structure category, or a noisy background category.

3. The method of claim 1 , wherein the plurality of imaging processing methods comprises at least one of: a filtering and thresholding method, a basic line detector method, a deep learning method, a Fourier-based analysis method, a wavelet-based analysis method, or a morphological component analysis method.

4. The method of claim 3, wherein the morphological component analysis method is based on a directional pattern in the image.

5. The method of claim 1 , wherein the determining the plurality of scores further comprises: determining an individual score of the plurality of scores for a respective one of the subset of the image processing methods based on a weighted sum of the plurality of metrics.

6. The method of claim 1 , wherein the plurality of metrics used for determining each of the plurality of scores comprises: a first metric based on an entropy measure of the image, a second metric based on an average difference between intra-region disparity and inter-region disparity in the image, and a third metric based on at least one of a loss of correlation, a luminance distortion, or a contrast distortion in the image.

7. The method of claim 1 , wherein the defect identified based on the recommended image processing method comprises at least one of a pit, a ridge, a scratch, a crack, or a pore.

8. The method of claim 1 , wherein at least one of the categorizing the image into the at least one of the plurality of manufacturing defect categories or the selecting the subset of the image processing methods are based on machine learning.

9. A computer-implemented method for detecting defects in manufacturing applications based on a manufacturing process driven image analysis, the method comprising: receiving, by a toolbox comprising instructions stored in a non-transitory memory of a computer system and executed by a processor of the computer system, an image of a product or product component; selecting, by the toolbox, a subset of image processing methods of a plurality of image processing methods to apply to the image based on a manufacturing process applied to the product or product component; initiating, by the toolbox, application of the selected subset of image processing methods; receiving, by the toolbox, results of the application of the selected subset of image processing methods; determining, by the toolbox, a recommendation score for each of the selected subset of image processing methods based on the results; selecting, by the toolbox, an image processing method of the subset of image processing methods based on the recommendation score associated with the image processing method; and providing, by the toolbox, the selected image processing method for use in defect detection for the product or product component.

10. The method of claim 9, further comprising: applying, by the toolbox, the selected image processing method to one or more images of the product or product component.11 . The method of claim 9, further comprising: initiating, by the toolbox, application of the selected image processing method to one or more images of the product or product component.

12. The method of claim 11 , wherein the initiating the selected image processing method comprises:issuing, by the toolbox, a command to a corresponding system that applies the selected image processing method.

13. The method of claim 9, further comprising: receiving, by the toolbox, an updated image of the product or product component; selecting, by the toolbox, a second subset of image processing methods of the plurality of image processing methods to apply to the updated image based on a manufacturing process applied to the product or product component; initiating, by the toolbox, application of the selected second subset of image processing methods; receiving, by the toolbox, results of the application of the selected second subset of image processing methods; determining, by the toolbox, a recommendation score for each of the selected second subset of image processing methods based on the results of application of the selected second subset of image processing methods; selecting, by the toolbox, a second image processing method of the second subset of image processing methods based on the recommendation score associated with the second image processing method; and providing, by the toolbox, the selected second image processing method for use in defect detection for the product or product component, wherein the selected image processing method and the selected second image processing method are different.

14. The method of claim 13, wherein the subset of image processing methods and the second subset of image processing methods are the same.

15. The method of claim 9, wherein the toolbox implements one or more machine learning models.

16. The method of claim 15, further comprising: tuning, by the toolbox, the one or more machine learning models based on feedback about previously selected image processing methods for images of products or product components.

17. A system for defect identification, the system comprising:a non-transitory memory; a processor; and a toolbox comprising instructions stored in the non-transitory memory that, when executed by the processor, causes the toolbox to: receive an image of a product or product component, select a subset of image processing methods of a plurality of image processing methods to apply to the image based on at least one of a manufacturing process or a manufacturing defect category associated with the product or product component, initiate application of the selected subset of image processing methods, receive results of the application of the selected subset of image processing methods, determine a recommendation score for each of the selected subset of image processing methods based on the results, select an image processing method of the subset of image processing methods based on the recommendation score associated with the image processing method, and provide the selected image processing method for use in detecting a defect in the product or product component.

18. The system of claim 17, wherein the defect comprises at least one of pits, ridges, scratches, cracks, or pores.

19. The system of claim 17, wherein the toolbox further applies the selected image processing method to one or more images of the product or product component.

20. The system of claim 17, wherein the toolbox further initiates application of the selected image processing method to one or more images of the product or product component.

Images